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Highlights 


Why  GAO  Did  This  Study 

The  Department  of  the  Navy  (DON), 
a  major  component  of  the 
Department  of  Defense  (DOD),  has 
launched  its  Next  Generation 
Enterprise  Network  (NGEN)  program 
to  replace  the  Navy  Marine  Corps 
Intranet  (NMCI)  program.  NGEN 
capabilities,  such  as  secure  transport 
of  voice  and  data,  data  storage,  and  e- 
mail,  are  to  be  incrementally  acquired 
through  multiple  providers.  As 
planned,  the  first  increment  is 
expected  to  provide  comparable 
NMCI  capabilities,  additional 
information  assurance,  and  greater 
DON  network  control,  at  a  cost  of 
about  $50  billion  through  fiscal  year 
2025.  Given  the  size,  importance,  and 
complexity  of  NGEN,  GAO  was  asked 
to  determine  whether  DON  has 
sufficiently  analyzed  alternative 
acquisition  approaches  and  has  a 
reliable  schedule  for  executing  the 
program,  and  whether  program 
acquisition  decisions  have  been 
performance-  and  risk-based.  To  do 
this,  GAO  reviewed  the  NGEN 
analysis  of  alternatives,  integrated 
master  schedule,  and  key  milestone 
decisions. 

What  GAO  Recommends 

GAO  is  recommending  that  DOD  limit 
further  investment  in  NGEN  until  it 
conducts  an  interim  review  to 
reconsider  the  selected  acquisition 
approach  and  addresses  issues 
discussed  in  this  report.  In  its 
comments,  DOD  stated  that  it  did  not 
concur  with  the  recommendation  to 
reconsider  its  acquisition  approach; 
GAO  maintains  that  without  doing  so, 
DOD  cannot  be  sure  it  is  pursuing  the 
most  cost-effective  approach. 


View  GAO-1 1-150  or  key  components. 

For  more  information,  contact  Valerie  C. Melvin 
at  (202)  512-6304  or  melvinv@gao.gov. 


What  GAO  Found 

DON  did  not  sufficiently  analyze  alternative  acquisition  approaches  for  NGEN 
because  the  alternatives  analysis  contained  key  weaknesses,  and  none  of  the 
alternatives  assessed  match  the  current  acquisition  approach.  Specifically,  the 
cost  estimates  for  the  respective  alternatives  were  not  reliable  because  they 
were  not  substantially  accurate,  and  they  were  neither  comprehensive  nor 
credible.  Further,  the  operational  effectiveness  analysis,  the  other  key  aspect 
of  an  analysis  of  alternatives,  did  not  establish  and  analyze  sufficient 
measures  for  assessing  each  alternative’s  ability  to  achieve  program  goals  and 
deliver  program  capabilities.  Moreover,  the  acquisition  approach  that  DON  is 
actually  pursuing  was  not  one  of  the  alternatives  assessed  in  the  analysis,  and 
it  is  riskier  and  potentially  costlier  than  the  alternatives  analyzed  because  it 
includes  a  higher  number  of  contractual  relationships.  According  to  program 
officials,  the  analysis  reflects  the  most  that  could  be  done  in  the  time  that  was 
available  to  complete  it,  and  they  do  not  view  the  alternative  selected  as 
materially  different  from  the  assessed  alternatives,  even  though  it  is  about 
$4.7  billion  more  costly.  (See  table  for  comparison  of  alternatives.) 


Comparison  of  NGEN  Alternative  Approaches 

Status  quo 

Alt.  2 

Alt.  3  variant 

Alt.  3 

Current  approach 

Contractual 

relationships 

3 

3 

10 

15 

21 

Estimated  cost3 

$10.3 

$10.8 

$10.8 

$10.7 

$15.6 

Relative  risk 

Least 

More 

Greater 

Greatest 

Undetermined 

Sources:  DON  data  (status  quo  and  alternatives  2,  3  variant,  and  3);  GAO  analysis  of  DON  data  (current  approach). 

Tiscal  year  201 1  -201 5  in  billions. 


DON  does  not  have  a  reliable  schedule  for  executing  NGEN.  Only  two  of  the 
four  subschedules  that  GAO  reviewed,  each  of  which  help  form  the  master 
schedule,  adequately  satisfied  any  of  the  nine  practices  that  are  associated 
with  developing  and  maintaining  a  reliable  schedule.  These  weaknesses  have 
contributed  to  delays  in  key  program  milestones.  During  the  course  of  GAO’s 
review,  DON  stated  that  action  was  taken  to  address  some,  but  not  all,  of 
these  weaknesses.  According  to  program  officials,  schedule  estimating  was 
constrained  by  staffing  limitations. 

NGEN  acquisition  decisions  were  not  always  performance-  and  risk-based.  In 
particular,  the  program  was  approved  in  the  face  of  known  performance 
shortfalls  and  risks.  For  example,  the  program  was  approved  at  a  key 
acquisition  review  despite  the  lack  of  defined  requirements,  which  was 
recognized  as  a  risk  that  would  impact  the  completion  of  other  key 
documents,  such  as  the  test  plan.  This  risk  was  later  realized  as  a  critical 
issue.  According  to  program  officials,  the  decisions  to  proceed  were  based  on 
their  view  that  they  had  sufficiently  mitigated  known  risks  and  issues. 

Collectively,  these  weaknesses  mean  that  DON  does  not  have  a  sufficient 
basis  for  knowing  that  it  is  pursuing  the  best  approach  for  acquiring  NGEN 
capabilities  and  the  program’s  cost  and  schedule  performance  is  unlikely  to 
track  to  estimates. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


March  11,  2011 

The  Honorable  Claire  McCaskill 
Chairman 

The  Honorable  Kelly  Ayotte 
Ranking  Member 

Subcommittee  on  Readiness  and  Management  Support 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Richard  Burr 
United  States  Senate 

The  Department  of  the  Navy  (DON)  established  the  Next  Generation 
Enterprise  Network  (NGEN)  program  in  2007  to  replace  the  Navy  Marine 
Corps  Intranet  (NMCI),  which  is  provided  through  a  DON-wide  network 
services  contract.  As  envisioned,  NGEN  capabilities,  including  secure 
transport  of  voice  and  data,  data  storage,  and  e-mail,  are  to  be 
incrementally  acquired  through  multiple  providers.  The  first  increment  is 
to  include  capabilities  comparable  to  NMCI,  as  well  as  enhanced 
information  assurance  and  increased  government  control  over  network 
operations.  To  date,  NGEN  has  reportedly  spent  about  $432  million  on 
work  associated  with  the  transition  from  NMCI.  The  first  increment  is  to 
be  fully  operational  in  March  2014  and  is  to  cost  approximately  $50  billion 
through  2025. 

As  agreed,  our  objectives  were  to  determine  whether  (1)  DON  has 
sufficiently  analyzed  alternative  approaches  for  acquiring  NGEN,  (2)  DON 
has  a  reliable  schedule  for  executing  NGEN,  and  (3)  acquisition  decisions 
have  been  performance-  and  risk-based.  To  accomplish  our  objectives,  we 
analyzed  the  NGEN  alternatives  analysis  report  and  underlying  support, 
the  program’s  integrated  master  schedule,  program  performance 
assessments  and  risk  reports,  and  executive  acquisition  decision  briefings 
and  meeting  minutes,  among  other  things.  We  conducted  this  performance 
audit  from  October  2009  to  February  2011,  in  accordance  with  generally 
accepted  government  auditing  standards.  Those  standards  require  that  we 
plan  and  perform  the  audit  to  obtain  sufficient,  appropriate  evidence  to 
provide  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our 
audit  objectives.  We  believe  that  the  evidence  obtained  provides  a 
reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit 
objectives.  Additional  details  on  our  objectives,  scope,  and  methodology 
are  discussed  in  appendix  I. 
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Background 


DON  is  a  major  component  of  the  Department  of  Defense  (DOD), 
consisting  of  the  Navy  and  the  Marine  Corps.  It  is  a  large  and  complex 
organization,  whose  primary  mission  is  to  organize,  train,  maintain,  and 
equip  combat-ready  naval  forces  capable  of  winning  wars,  deterring 
aggression  by  would-be  foes,  preserving  freedom  of  the  seas,  and 
promoting  peace  and  security.  To  support  this  mission,  DON  performs  a 
variety  of  interrelated  and  interdependent  information  technology  (In¬ 
dependent  functions.  In  fiscal  year  2010,  DON’s  IT  budget  was 
approximately  $7.4  billion,  for  971  investments.  NGEN  is  one  such  system 
investment. 


Overview  of  NGEN  NGEN  is  to  provide  secure  data  and  IT  services,  such  as  data  storage,  e- 

mail,  and  video-teleconferencing,  to  the  Navy  and  the  Marine  Corps. 

NGEN  is  also  intended  to  provide  the  foundation  for  DON’s  future  Naval 
Networking  Environment.1  DON  is  acquiring  NGEN  through  multiple 
providers  (contractors)  to  replace  and  improve  the  enterprise  network  and 
services  provided  by  NMCI.  It  is  to  be  developed  incrementally,  with  the 
first  increment  to  provide  comparable  NMCI  capabilities,2  additional 
information  assurance,  and  increased  government  control  of  the  network. 
Future  increments  have  yet  to  be  defined.  The  program’s  preliminary  life 
cycle  cost  estimate  (through  fiscal  year  2025)  for  the  first  increment  is 
about  $50  billion.  As  of  September  30,  2010,  the  NGEN  program  had 
reportedly  spent  about  $432  million. 

To  bridge  the  time  frame  between  the  end  of  the  NMCI  contract  and  the 
full  transition  to  NGEN,  DON  awarded  a  $3.7  billion  continuity  of  services 
contract  in  July  2010  to  the  NMCI  service  provider,  Hewlett  Packard 
Enterprise  Services.  In  addition  to  providing  continuity  of  network 
services,  the  contract  includes  transition  services  and  transfer  to  DON  of 


'The  Naval  Networking  Environment  is  to  be  an  iterative  set  of  integrated,  phased 
programs  that  share  a  common  enterprise  architecture  and  standards.  It  includes  NGEN 
and  the  Consolidated  Afloat  Networks  and  Enterprise  Services  program,  among  others,  and 
is  to  be  in  place  by  2016. 

2NMCI  provides  about  382,000  workstations  to  approximately  700,000  users  across  2,500 
Navy  and  Marine  Corps  locations  around  the  world.  NMCI  is  composed  of  transport 
infrastructure,  such  as  cables,  routers,  and  switches;  end-user  equipment,  such  as 
computers,  monitors,  and  keyboards;  and  software.  It  provides,  among  other  things,  data 
storage,  e-mail,  transport  of  voice  and  data,  and  video-teleconferencing. 


Page  2 


GAO-11-150  Information  Technology 


NMCI  infrastructure  and  intellectual  property,3  as  the  NGEN  contracts  are 
to  require  use  of  the  NMCI  infrastructure  and  access  to  processes, 
procedures,  and  technical  data.  The  continuity  of  services  contract  is 
scheduled  to  run  from  October  2010  through  April  2014. 

To  reduce  the  risk  during  the  transition  period  from  NMCI  to  NGEN,  DON 
is  currently  performing  eight  early  transition  activities.  The  activities  are 
discrete  efforts  intended  to  establish  government  management 
capabilities,  allow  for  greater  participation  in  operational  decisions,  and 
help  expedite  the  transition  time.  Table  1  describes  each  of  these 
activities. 


Table  1 :  Early  Transition  Activities 


Dollars  in  millions 


Early  transition  activity 

Start  date 

End  date 

Cost 

Description 

Information  Technology  Service 
Management  Process  Development 

October  2008 

May  2011 

$20.5 

Develop  Information  Technology  Infrastructure 
Library3  version  3-based  service  strategies, 
processes,  and  procedures. 

Contractor  Technical  Representative 
Workforce  Reconstitution 

J  anuary  2009 

April  2011 

$3.3 

Conduct  job  task  analysis  and  assess  learning 
tools  for  contractor  technical  representatives; 
develop  enterprise-wide  position  descriptions  and 
occupational  standards  for  training,  advancement, 
criteria,  and  performance  objectives. 

Comprehensive  Facilities  and 
Infrastructure  Inventory 

J  anuary  2009 

December 

2010 

$12.0 

Evaluate,  analyze,  and  validate  current  NMCI 
infrastructure  inventory  consisting  of  technical 
data,  assets,  configuration  items,  and  system 
components. 

Defense  Information  System  Network 
Core  Extension  Phase  1  and  Maritime 
Operation  Center  Implementation 

April  2008 

May  2011 

$6.7 

Bring  consistent  wide  area  network  connectivity 
from  the  Defense  Information  System  Network  to 
eight  major  nodes  at  fleet  headquarters  (Norfolk, 
Virginia,  and  Pearl  Harbor,  Hawaii). 

Global  Network  Operations  Command 
and  Control  Workforce  Establishment 

October  2008 

J  uly  2011 

$14.3 

Develop  the  personnel,  processes,  and  tool 
requirements,  and  organizational  analysis  and 
alignment. 

Wide  Area  Network  and  Enterprise 
Services  Upgrade 

April  2009 

August  2012 

$46.0 

Demonstrate  network  operational  control 
capability  and  validate  the  NGEN  System  Design 
baseline  through  early  implementation. 

3The  NMCI  contractor  will  deliver  in  place  the  NMCI  infrastructure  and  provide  DON  with  a 
Government  Purpose  Rights  license  for  NMCI  technical  data,  computer  software,  and 
computer  software  documentation. 
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Dollars  in  millions 


Early  transition  activity 

Start  date 

End  date  Cost  Description 

Enterprise  Tools  Strategy  and 
Implementation/Integration 

April  2009 

May  2011  $56.6  Analyze  current  tool  capabilities  to  support 

information  technology  service  management 
processes,  and  develop  design  requirements  and 
tool  integration  specifications. 

Nonclassified  Internet  Protocol  Router 
Network  Migration  -  Marine  Air- 
Ground  Task  Force  IT  SupportCenters 
East  Pilot 

October  2009 

August2011  $12.9  Assess  the  transition  of  base  area  network,  local 

area  network,  and  end-user  equipment  for  about 
1,200  users  from  the  continuity  of  services 
contract  to  the  government-owned/government- 
operated  NGEN  environment. 

Source:  GAO  analysis  of  DON  data. 

information  Technology  Infrastructure  Library  is  a  framework  intended  to  align  an  organization's  IT 
services  with  its  organizational  needs.  It  consists  of  service  strategy,  design,  transition,  operation  and 
continual  improvement  best  practice  guidance. 

To  deliver  NGEN  capabilities,  DON  plans  to  award  five  contracts.4  See 
table  2  for  a  description  of  these  contracts. 

Table  2:  Planned  NGEN  Program  Contracts 

Contract 

Award  date 

Purpose 

Independent  Security  Operations, 
Oversight  and  Assessment  Support 

April  2011 

Provide  independent  third-party  security  assessments  of  NMCI  and 
NGEN.  The  contract  is  planned  to  include  the  option  to  support  all  DON 
networks  within  the  naval  network  environment. 

Transport  Services 

May  2012 

P  rovide  for  the  operation  and  sustainment  of  the  transport 
infrastructure,  associated  services,  and  level-of-effort  support  for  those 
services.  The  contract  is  planned  to  include  technology  refresh  of  cable 
plant,  routers,  and  switches;  some  leasehold  improvements;  and 
moveable  infrastructure  associated  with  local  network  operations. 

End-User  Hardware 

December  2011  Provide  end-user  equipment  such  as  computers,  monitors,  and 
keyboards. 

Enterprise  Software  Licenses 

March  2012 

Provide  software  licenses  to  meet  DON-wide  requirements. 

Enterprise  Services/Service  Coordination  August  2012 

Provide  the  Enterprise  Service  Desk,  seatservices  supporting  end-user 
devices,  and  data  center  services  such  as  storage  and  e-mail  along 
with  hardware  and  software  specific  to  Enterprise  Services  that  are  not 
covered  underthe  End-User  Hardware  and  Enterprise  Software 

Licenses  contracts.  As  the  service  coordinator,  this  contract  is  planned 
to  cover  coordination  across  all  NGEN  vendors  for  the  successful 
delivery  of  NGEN  services. 

Source:  GAO  analysis  of  DON  data. 


4In  addition  to  these  contracts,  DON  plans  to  enter  into  an  interagency  agreement  with  the 
Defense  Information  Systems  Agency  to  provide  wide  area  network  services. 
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According  to  the  NGEN  Acquisition  Strategy,  DON  plans  to  complete  the 
Marine  Corps’  initial  transition  to  NGEN  in  January  2012  and  final 
transition  in  February  2013.  The  Navy’s  initial  and  final  transition  to  NGEN 
are  scheduled  to  be  completed  in  December  2012  and  March  2014, 
respectively.5 

NGEN  Oversight  and 
Management  Roles  and 
Responsibilities 

To  manage  the  acquisition  and  deployment  of  NGEN,  DON  established  a 
program  management  office  within  the  Program  Executive  Office  for 
Enterprise  Information  Systems.  The  program  office  manages  the 
program’s  cost,  schedule,  and  performance  and  is  responsible  for  ensuring 
that  the  program  meets  its  objectives.  In  addition,  various  DOD  and  DON 
organizations  share  program  oversight  and  review  responsibilities.  Table  3 
lists  key  entities  and  their  roles  and  responsibilities. 

Table  3:  Organizations  Responsible  for  NGEN  Acquisition  Oversight  and  Management 

Entity 

Roles  and  responsibilities 

Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics 

Serves  as  the  milestone  decision  authority,  with  overall  responsibility  for  the  program,  to 
include  approving  the  program  to  proceed  through  its  acquisition  cycle  on  the  basis  of,  for 
example,  the  acquisition  strategy,  an  independently  evaluated  economic  analysis,  and  the 
Acquisition  Program  Baseline. 

Director,  Cost  Assessment  and  Program 
Evaluation,  Office  of  the  Secretary  of 
Defense  (OSD  CAPE) 

Issued  NGEN  analysis  of  alternatives  (AOA)  guidance,  reviewed  the  AOA  study  plan,  and 
approved  the  AOA  results.  Verifies  and  validates  the  reliability  of  cost  and  benefit 
estimates  found  in  economic  analyses. 

NGEN  AOA  Advisory  Group 

Oversaw  the  conduct  ofthe  NGEN  AOA.  Led  by  the  DON  Chief  Information  Officer. 

Assistant  Secretary  of  the  Navy, 

Research,  Development,  and  Acquisition 

Serves  as  DON'S  acquisition  oversight  organization  forthe  program,  to  include 
enforcement  of  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
policies  and  procedures.  Determines  when  all  key  milestones  are  ready  to  be  submitted 
to  the  milestone  decision  authority. 

Chief  of  Naval  Operations  and 
Commandant  of  the  Marine  Corps 

Serve  together  as  resource  sponsors  for  NGEN  requirements  and  funding. 

5NGEN  program  documentation  defines  initial  transition  for  the  Navy  as  execution  and 
operation  of  NGEN  services  for  5  percent  of  the  Navy  management  domain,  and  for  the 
Marine  Corps  as  execution  and  operation  of  5  percent  of  Marine  Corps  seats.  Final 
transition  is  defined  as  complete  when  increased  government  operational  control  and 
visibility  are  established  and  test  and  evaluation  requirements  are  verified  and  validated  for 
80  percent  of  the  Navy  management  domain  and  when  80  percent  of  Marine  Corps  seats  are 
transitioned  and  increased  operation  control  of  the  network  and  visibility  activities  are 
implemented  at  the  global,  regional,  and  local  level. 
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Entity 

Roles  and  responsibilities 

Department  of  the  Navy,  Program 

Executive  Office  for  Enterprise  Information 
Systems 

Oversees  a  portfolio  of  large-scale  projects  and  programs  designed  to  enable  common 
business  processes  and  provide  standard  capabilities.  Reviews  the  acquisition  strategy, 
economic  analysis,  and  the  Acquisition  Program  Baseline  prior  to  approval  by  the 
milestone  decision  authority. 

Department  of  the  Navy  Chief  Information 
Officer  (CIO) 

Supports  DON'S  planning,  programming,  budgeting,  and  execution  processes  by  ensuring 
that  the  program  has  achievable  and  executable  goals  and  conforms  to  financial 
management  regulations,  and  DON,  DOD,  and  federal  IT  policies  in  several  areas  (e.g ., 
security,  architecture,  and  investment  management);  works  closely  with  the  program 
office  during  milestone  review  assessments.  Oversaw  NGEN  AOA,  which  was  led  by  the 
Center  for  Naval  Analyses  and  supported  by  Deloitte  Consulting. 

NGEN  Program  Management  Office 

Performs  day-to-day  program  management  and,  as  such,  is  the  single  point  of 
accountability  for  managing  the  program's  objectives. 

Space  and  Naval  Warfare  Systems 
Command 

Reviewed  the  NGEN  AOA  cost  analysis.  Reviews  the  NGEN  program  life  cycle  cost 
estimates  and  economic  analysis.  Serves  jointly  with  the  Marine  Corps  Systems 

Command  as  the  NGEN  technical  authority. 

Marine  Corps  Systems  Command 

Serves  jointly  with  the  Space  and  Naval  Warfare  Systems  Command  as  the  NGEN 
technical  authority. 

Source:  GAO  analysis  of  DON  data. 

Overview  of  DOD  and 

DON  Acquisition 

Processes  and  NGEN 

Status 

NGEN  is  subject  to  both  Office  of  the  Secretary  of  Defense  (OSD)  and 

DON  Major  Automated  Information  System  (MAIS)  acquisition  policy  and 
guidance,6  which  require  it  to  comply  with  Defense  Acquisition  System 
(DAS)7  requirements.  According  to  these  requirements,  all  MAIS  programs 
require  a  Materiel  Development  Decision  prior  to  entering  the  first  DAS 
phase.  In  making  this  decision,  the  milestone  decision  authority  is  to 
review  the  Initial  Capabilities  Document,  which  defines  operational  goals 
and  needed  capabilities,  and  authorizes  the  phase  in  which  the  program  is 
to  enter  the  DAS.  The  system  consists  of  five  key  program  life  cycle  phases 
and  three  related  milestone  decision  points.  Table  4  provides  a  description 
of  each  DAS  phase. 

department  of  Defense  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System, 
Dec.  8,  2008,  and  Secretary  of  the  Navy  Instruction  5000.2D,  Implementation  and 
Operation  of  the  Defense  Acquisition  System  and  the  Joint  Capabilities  Integration  and 
Development  System  (Oct.  16,  2008). 

The  DAS  five-phase  acquisition  process  is  intended  to  translate  mission  needs  and 
requirements  into  stable,  affordable,  and  well-managed  acquisition  programs. 
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Table  4:  Description  of  DAS  Phases 

Phase 

Description 

Materiel  Solution  Analysis 

Prior  to  entering  this  phase  the  milestone  decision  authority  completes  a  Materiel  Development 
Decision.  The  purpose  of  the  phase  is  to  assess,  through  an  AO  A,  potential  solutions  to  satisfy 
an  approved  capability  need. 

Technology  Development 

Priorto  entering  this  phase,  the  milestone  decision  authority  conducts  a  Milestone  A  decision 
review  of  the  proposed  solution  and  a  draft  Technology  Development  Strategy.  The  purpose  of 
the  phase  is  to  determine  the  appropriate  set  of  technologies  to  be  integrated  into  the  solution 
and  to  refine  user  requirements. 

Engineering  and  Manufacturing 
Development 

M  ost  acquisition  programs  enter  the  DAS  in  this  phase.  P  rior  to  entering  the  phase,  the 
milestone  decision  authority  completes  a  Milestone  B  review  to  approve  the  Acquisition 

Strategy  and  Acquisition  Program  Baseline.3  The  purpose  of  the  phase  is  to  develop  a  system 
or  system  increment.  The  phase  has  two  major  efforts:  Integrated  System  Design  and  System 
Capability  and  Manufacturing  Process  Demonstration. 

Integrated  System  Design: This  effort  includes  a  post-Critical  Design  Review  assessment  of 
design  maturity,  and  issues  and  risks  that  could  result  in  a  breach  to  the  Acquisition  P  rogram 
Baseline  or  substantively  impact  cost,  schedule,  or  performance. 

System  Capability  and  Manufacturing  Process  Demonstration :  This  effort  is  to  demonstrate  the 
ability  of  the  system  to  operate  according  to  the  approved  Key  Performance  Parameters11  and 
ends  when  the  system  or  system  increment  meets  approved  requirements. 

Production  and  Deployment 

Priorto  entering  this  phase,  the  milestone  decision  authority  conducts  a  Milestone  C  review  of 
the  Capability  Production  Documentand  the  Test  and  Evaluation  Master  Plan,  among  other 
things,  and  authorizes  limited  deployment  to  support  operational  testing.  The  purpose  ofthe 
phase  is  to  achieve  an  operational  capability  that  satisfies  the  mission  needs,  verified  through 
independent  operational  test  and  evaluation,  and  to  implement  the  system  at  all  applicable 
locations.  During  this  phase,  the  milestone  decision  authority  also  conducts  a  Full  Deployment 
Decision  Review. 

Operations  and  Support 

The  purpose  of  this  phase  is  to  operationally  sustain  the  system  in  the  most  cost-effective 
manner  over  its  life  cycle. 

Source:  GAO  analysis  of  DOD  documentation. 


’According  to  DOD's  acquisition  guidebook,  an  Acquisition  P  rogram  Baseline  is  a  program  manager's 
estimated  cost,  schedule,  and  performance  goals.  Goals  consist  of  objective  values,  which  represent 
what  the  user  desires  and  expects,  and  threshold  values,  which  represent  acceptable  limits.  When 
the  program  manager  determines  that  a  current  cost,  schedule,  or  performance  threshold  value  will 
not  be  achieved,  the  milestone  decision  authority  must  be  notified,  and  a  new  baseline  developed, 
reviewed  by  decision  makers  and,  if  the  program  is  to  continue,  approved  by  the  milestone  decision 
authority. 

bKey  Performance  Parameters  are  attributes  or  characteristics  of  a  system  thatare  considered  critical 
or  essential  to  the  development  of  an  effective  military  capability. 

In  addition  to  Defense  Acquisition  System  requirements,  according  to  DON 
guidance  and  policy,  all  DON  MAIS  and  pre-MAIS  programs  are  required  to 
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go  through  a  “Two-Pass/Six-Gate”  acquisition  review  process.8  The  first 
pass,  which  consists  of  Gates  1  through  3,  is  focused  on  requirements 
development  and  validation  and  is  led  by  the  Chief  of  Naval  Operations  or 
the  Commandant  of  the  Marine  Corps.  The  second  pass,  which  consists  of 
Gates  4  through  6,  is  focused  on  developing  and  delivering  a  solution  via 
systems  engineering  and  acquisition  and  is  led  by  the  Assistant  Secretary 
of  the  Navy  (Research,  Development,  and  Acquisition).  In  addition  to 
meeting  specific  criteria  for  passing  a  given  gate  and  proceeding  to  the 
next  gate,  all  gate  reviews  are  to  consider  program  health  (i.e.,  satisfactory 
cost  and  schedule  performance,  known  risks,  and  budget  adequacy)  in 
deciding  whether  to  proceed.  Table  5  lists  the  key  purpose  of  each  gate 
review. 


Table  5:  Purpose  of  DON  Gate  Reviews 

Gate 

Purpose 

1 

To  approve  the  Initial  Capabilities  Document  and  to  validate  AOA  guidance  and 
assumptions. 

2 

To  review  the  AOA  and  approve  the  preferred  alternatives  resulting  from  the 
analysis. 

3 

To  authorize  the  Capability  Development  Document,  which  defines  the  system's 
Key  Performance  Parameters  and  includes  information  necessary  to  develop  an 
affordable  system  or  system  increment,  to  be  submitted  toj  oint  Staff  for  review, 
and  to  approve  the  Concept  of  Operations. 

4 

To  approve  the  System  Design  Specification,  which  specifies  the  system 
requirements  and  is  derived  from  the  Capability  Development  Document. 

5 

T o  approve  the  release  of  a  request  for  proposal  to  industry. 

6 

To  review  overall  program  health,  following  award  of  a  contract  and  satisfactory 
completion  of  an  Integrated  Baseline  Review.3  Also,  to  approve  the  Capability 
Production  Document  and  program  health  prior  to  and  after  Milestone  C  and  the 
Full  Deployment  Decision  Review. 

Source:  GAO  analysis  of  DON  documentation. 


sAn  Integrated  Baseline  Review  is  performed  to  obtain  stakeholder  agreement  on  a  contractor's 
Performance  Measurement  Baseline,  which  represents  the  cumulative  value  of  planned  work  and 
serves  as  the  baseline  against  which  variances  are  calculated. 

The  DAS  and  DON  acquisition  phases  and  decision  points  for  MAJS 
programs  are  illustrated  in  figure  1. 


Secretary  of  the  Navy  Instruction  5000. 2D,  Implementation  and  Operation  of  the  Defense 
Acquisition  Si/stem.  and  the  Joint  Capabilities  Integration  and  Development  System 
(Oct.  16,  2008). 
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Figure  1 :  Simplified  View  of  OSD  DAS  and  DON  Gate  Review  Process 
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Source:  GAO  analysis  of  DOD  and  DON  policy  and  guidance. 


Note:  MDD  =  M  ateriel  Development  Decision;  CDR  =  C  ritical  Design  Review;  FDDR  =Full 
Deployment  Decision  Review;  ICD  =  Initial  Capabilities  Document;  AOA  =  a na lysis  of  alternatives; 
CDD  =  C  apability  Development  Document;  SDS  =System  Design  Specification;  RFP  =requestfor 
proposal;  IBR  integrated  Baseline  Review;  and  CPD  =  C apability  Production  Document. 

As  depicted  in  figure  2,  DON  completed  a  Gate  3  review  of  NGEN 
requirements  in  April  2008. 9  In  April  2009,  the  DON  CIO  completed  the 
AOA  for  NGEN  increment  1,  and  at  the  Gate  2  review  the  same  month,  the 
Deputy  Chief  of  Naval  Operations  (Integration  of  Capabilities  and 
Resources)  and  the  Deputy  Marine  Corps  Commandant  for  Programs  and 
Resources  approved  the  AOA  to  be  submitted  to  the  NGEN  AOA  Advisory 
Group.10  The  advisory  group  subsequently  approved  the  analysis  and 
forwarded  it  in  April  2009  to  OSD  Cost  Assessment  and  Program 
Evaluation  (CAPE),  which  approved  it  in  December  2009.  DON  conducted 
a  Gate  4  review  of  its  System  Design  Specification  in  November  2009,  and 
a  Gate  5  review  of  its  Transport  Services  request  for  proposal  in  October 
2010.  DON  plans  to  conduct  a  Gate  6  review  in  July  2011. 

In  May  2010,  the  USD  (AT&L)  completed  the  NGEN  Materiel  Development 
Decision,  which  designated  the  first  increment  of  NGEN  as  a  MAIS  and 


9The  gate  process  had  not  been  established  at  the  time.  DON  reported  that  Gate  1  was 
nominally  held  in  a  series  of  Chief  of  Naval  Operations  Executive  Board  meetings  during 
the  winter  of  2007/2008. 

10Gate  2  was  initially  waived  because  DON’s  original  acquisition  approach  was  to  continue 
to  use  the  existing  NMCI  technology  and,  therefore,  a  traditional  analysis  of  alternative 
technologies  was  not  planned.  Subsequently,  OSD  identified  the  program  as  a  pre-Major 
Automated  Information  System  acquisition,  and  as  a  result  an  analysis  of  acquisition 
alternatives  was  directed.  A  Gate  2  review  was  conducted  to  review  the  AOA. 
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authorized  the  program  to  enter  the  DAS  in  the  production  and 
deployment  phase.  A  Milestone  C  review  is  currently  planned  for  August 
2011.  In  June  2010,  the  USD  (AT&L)  approved  the  current  acquisition 
approach. 


Figure  2:  NGEN  Review  Schedule 


Note:  MDD  =  M ateriel  Development  Decision;  AOA  =  a na lysis  of  alternatives;  SDS  =System  Design 
Specification;  RFP  =  request  for  proposal;  and  CPD  =  Capability  Production  Document. 


Current  NGEN 
Acquisition  Approach 
Is  Not  Grounded  in  a 
Reliable  AOA 


An  AOA  is  intended  to  help  identify  the  most  promising  acquisition 
approach  by  comparing  alternative  solutions’  costs  and  operational 
effectiveness.  The  NGEN  AOA  contained  key  weaknesses  in  its  cost 
estimates  and  operational  effectiveness  analysis  that  impaired  its  ability  to 
inform  investment  decision  making.  Further,  none  of  the  alternatives  in 
this  analysis  match  the  current  acquisition  approach,  and  these  differences 
have  not  been  analyzed  to  determine  the  breadth  of  risk  that  exists. 
According  to  DON  officials,  the  AOA  reflects  the  most  that  could  be 
accomplished  in  the  time  available  to  meet  an  imposed  deadline.  In 
addition,  OSD  officials  stated  that  the  differences  between  the  current 
approach  and  the  alternatives  that  were  assessed  are,  in  their  view,  not 
significant.  However,  the  current  approach  is  estimated  to  cost  at  least 
$4.7  billion  more  than  any  of  the  AOA  alternatives.  Without  sufficient 
information  to  understand  the  differences  in  the  relative  costs  and 
operational  effectiveness  among  alternatives,  decision  makers  lack 
assurance  that  their  selected  approach  is  the  most  promising  and  cost- 
effective  course  of  action. 
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NGEN  AOA  Was  Not 

Based  on  Reliable  Cost 
Estimates 

According  to  relevant  guidance, 11  a  key  component  of  an  AOA  is  a  cost 
analysis  that  provides  for  cost  estimates  of  each  alternative.  As  such,  cost 
estimates  should  be  reliable  in  order  to  provide  the  basis  for  informed 
investment  decision  making,  realistic  budget  formulation,  meaningful 
progress  measurement,  and  accountability  for  results.  Our  research  has 
identified  four  characteristics  of  a  high-quality,  reliable  cost  estimate:  well- 
documented,  comprehensive,  accurate,  and  credible. 12 

The  NGEN  AOA  assessed  four  alternatives. 13  All  alternatives  were 
assumed  to  deliver  the  same  NMCI  capabilities  and  the  technology  across 
alternatives  was  assumed  to  be  substantially  the  same.  The  primary 
differences  among  the  alternatives  were  how  NGEN  was  to  be  acquired, 
managed,  and  operated.  Table  6  below  provides  a  description  of  each 
alternative. 

Table  6:  Summary  of  Four  Alternatives  in  the  NGEN  AOA 

Alternative 

Description 

Alternative  1  (status  quo) 

A  recompete  of  the  NMCI  contract,  in  which  the  contractor  was  to  be  responsible  for  end- 
to-end  integration  of  services  and  control  of  the  network. 

Alternative  2  (enhanced  status  quo) 

A  single  contract  similar  to  NMCI  but  with  upgraded  terms  and  conditions  that  were  to 
address  known  NMCI  deficiencies.3  The  contractor  was  to  be  responsible  for  end-to-end 
integration  of  services,  and  the  government  would  control  the  network. 

Alternative  3  Variant  (3V)  (segmented11 
approach) 

Multiple  contracts  with  different  vendors.  The  government  was  to  be  responsible  for  end- 
to-end  integration  of  services  and  control  of  the  network. 

Alternative  3  (segmented  approach) 

A  greater  number  of  contracts  than  alternative  3V.  The  government  was  to  be 
responsible  for  end-to-end  integration  of  services  and  control  of  the  network. 

Source:  NGEN  Analysis  of  Alternatives  report,  Center  for  Naval  Analyses,  April  2009. 


’See,  for  example,  GAO,  Information  Technology:  DOD  Needs  to  Ensure  That  Navy  Marine  Corps 
Intranet  Program  Is  Meeting  Goals  and  Satisfying  Customers,  GAO-07-51  (Washington,  D.C Dec.  8, 
2006),  for  information  on  NMCI  deficiencies. 

bSegments  are  components  of  IT  services  to  be  delivered  for  NGEN  through  multiple  contracts. 


nGAO,  GAO  Cost  Estimating  and  Assessm  en  t  Guide:  Best  Practices  for  Developing  and 
Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009);  Defense 
Acquisition  University,  Defense  Acquisition  Guidebook,  Section  3.3  “Analysis  of 
Alternatives”  (accessed  Mar.  19,  2010). 

12GAO-09-3SP. 

13DON  officials  stated  that  the  NGEN  AOA  was  not  intended  to  be  a  traditional  analysis  to 
determine  a  system  solution;  rather,  it  was  an  analysis  of  alternative  acquisition 
approaches. 
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The  four  alternatives’  estimated  costs  for  increment  1  from  fiscal  year  2011 
to  fiscal  year  2015  ranged  from  $10.25  billion  (alternative  1)  to  $10.84 
billion  (alternatives  2  and  3V).  However,  the  estimates  were  not  reliable 
because  they  substantially  met  only  one  of  the  characteristics  of  reliable 
cost  estimates.  Specifically, 

•  The  AOA  cost  estimates  were  substantially  well-documented.  To  be  well- 
documented,  the  cost  estimates  should  state  the  purpose  of  the  estimate; 
provide  program  background,  including  a  system  description;  provide  a 
schedule  for  developing  the  estimates;  specify  the  scope  of  the  estimate 
(in  terms  of  time  and  what  is  and  is  not  included);  disclose  key  ground 
rules  and  assumptions,  data  sources,  calculations  performed  and  their 
results,  the  estimating  methodology  and  rationale,  and  the  results  of  a  risk 
analysis;  and  provide  a  conclusion  about  whether  the  cost  estimate  is 
reasonable.  Moreover,  this  information  should  be  captured  in  such  a  way 
that  the  data  used  to  derive  the  estimate  can  be  traced  to  their  sources. 
Finally,  the  cost  estimates  should  be  reviewed  and  accepted  by 
management. 

Although  the  AOA  did  not  sufficiently  document  the  schedule,  scope,  and 
results  of  the  risk  analysis,  it  defined  the  purpose  of  the  estimate;  provided 
program  background  (e.g.,  system  description);  and  disclosed  ground 
rules  and  assumptions,  data  sources,  calculations  performed  and  their 
results,  and  the  estimating  methodology.  Also,  the  data  used  to  derive  the 
estimates  were  captured  in  such  a  way  that  they  could  largely  be  traced  to 
their  sources,  and  the  final  AOA  was  reviewed  and  accepted  by  DON  and 
OSD  oversight  entities. 

•  The  AOA  cost  estimates  were  not  comprehensive.  To  be  comprehensive, 
the  cost  estimates  should  include  all  government  and  contractor  costs 
over  the  program’s  full  life  cycle,  from  program  inception  through  design, 
development,  deployment,  and  operation  and  maintenance  to  retirement. 
They  should  also  provide  sufficient  detail  and  reflect  all  cost-influencing 
ground  rules  and  assumptions. 

However,  the  cost  estimates  were  not  full  life  cycle  costs.  Instead,  they 
only  included  government  and  contractor  costs  for  a  5-year  period  from 
fiscal  year  2011  to  fiscal  year  2015,  covering  2  years  of  continued  NMCI 
services  with  the  current  provider,  2  years  of  transition  to  the  new 
provider(s),  and  1  year  of  NGEN  operation  and  maintenance.  DON  and 
OSD  CAPE  officials  attributed  this  to  the  assumption  that  NGEN 
increment  1  contracts  would  have  a  5-year  period  of  performance  and  that 
future  NGEN  increments  might  be  introduced  after  that  period. 
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Further,  while  the  estimates  were  based  on  a  cost  element  structure  that 
was  decomposed  to  a  sufficient  level  of  detail,  and  the  documentation 
largely  identified  ground  rules  and  assumptions,  the  cost  estimates  did  not 
reflect  all  assumptions  identified  in  the  AO  A,  such  as  schedule  and 
performance  risks  associated  with  (1)  implementing  IT  processes,  (2) 
expanding  the  government  workforce,  and  (3)  formulating  the  NGEN 
contracts.  These  were  significant  cost-influencing  risks  and  thus  should 
have  been  incorporated  into  the  estimates. 

•  The  AOA  cost  estimates  were  not  substantially  accurate.  To  be  accurate, 
the  cost  estimates  should  not  be  overly  conservative  or  optimistic  and 
should  be,  among  other  things,  based  on  an  assessment  of  the  most  likely 
costs,  and  adjusted  properly  for  inflation.  In  addition,  steps  should  be 
taken  to  minimize  mathematical  mistakes  and  to  ground  the  estimate  in 
documented  assumptions  that  can  be  verified  by  supporting  data  and  a 
historical  record  of  actual  cost  and  schedule  experiences  on  comparable 
programs. 

To  DON’s  credit,  the  cost  estimates  were  developed  based  on  NMCI 
historical  cost  data,  were  adjusted  properly  for  inflation,  contained  few 
mathematical  mistakes,  and  were  largely  grounded  in  documented 
assumptions.  However,  the  supporting  data  for  key  assumptions  were  not 
verified.  For  example,  all  estimates  assumed  that  transition  activity  costs 
would  amount  to  about  18  percent  of  the  estimated  cost  of  NGEN  in  its 
first  year  of  operation,  and  alternative  3’s  estimate  assumed  that  total  cost 
would  be  reduced  by  10  percent  due  to  increased  competition  from  its 
multicontract  approach.  However,  the  supporting  data  used  by  Deloitte 
Consulting  for  these  assumptions  were  not  provided  to  DON  or  the 
independent  government  review  team  for  verification  because  the  data 
were  proprietary  to  the  contractor.  Further,  NMCI  historical  data  were 
only  available  at  an  aggregate  level,  so  the  team  had  to  rely  on  subject- 
matter  experts  and  other  sources  to  estimate  costs  at  a  finer  level  of  detail. 

•  The  AOA  cost  estimates  were  not  credible.  To  be  credible,  the  cost 
estimates  should  discuss  any  limitations  in  the  analysis  due  to  uncertainty 
or  biases  surrounding  the  data  and  assumptions.  Major  assumptions 
should  be  varied  and  other  outcomes  computed  to  determine  how 
sensitive  the  estimate  is  to  changes  in  the  assumptions.  Risk  and 
uncertainty  inherent  in  the  estimate  should  be  assessed  and  disclosed. 
Further,  the  estimate  should  be  properly  verified  by,  for  example, 
comparing  the  results  with  an  independent  cost  assessment. 
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While  the  AOA  identified  limitations  in  the  cost  analysis,  such  as  the  use  of 
NMCI  data  that  did  not  reflect  prices  of  other  service  providers,  and 
evaluated  the  impact  on  costs  of  using  different  transition  timeline 
scenarios,  it  did  not  include  a  sensitivity  analysis  of  the  key  cost  driver 
(i.e.,  the  number  of  personnel  needed  to  manage  NGEN),  despite  concerns 
that  the  Navy’s  estimates  of  these  numbers  were  not  stabilized  at  the  time 
of  the  AOA.  In  addition,  while  each  cost  estimate  included  a  cost  risk 
analysis  based  on  the  quality  of  data  used,  there  were  discrepancies  in 
how  the  analysis  was  conducted  and  reported.  For  example,  the  cost  for 
local  area  network  facilities  was  estimated  based  on  the  contractor’s 
experience,  which  was  considered  by  the  cost  team  to  be  a  less  credible 
data  source,  but  it  was  scored  higher  on  their  risk  scale,  indicating  that  the 
data  source  was  more  credible.  Also,  schedule  and  performance  risks 
were  not  quantified  and  reflected  in  the  estimates,  which  is  significant 
because  a  qualitative  assessment  of  schedule  and  performance  risks 
among  alternatives  revealed  increased  risk  in  implementing  a  segmented 
approach.  If  such  risks  had  been  quantified  and  reflected  in  the  estimates, 
the  results  would  have  shown  higher  costs  associated  with  alternatives  3 
and  3V.  Nevertheless,  the  AOA  concluded  that  there  was  no  significant 
cost  difference  among  the  alternatives. 

In  addition,  the  cost  estimates  were  not  validated  by  the  independent  team 
responsible  for  reviewing  the  cost  analysis.  Specifically,  independent 
review  team  officials  told  us  that  they  participated  in  a  line-by-line  review 
of  the  cost  model  where  they  raised  comments  and  questions  to  the  cost 
team.  However,  about  69  percent  of  the  team’s  comments  were  not  fully 
addressed  and  included  notable  concerns,  such  as  the  questionable  use  of 
certain  industry-based  assumptions  that  may  not  be  comparable  to  a 
program  as  large  as  NGEN  or  to  the  government  environment. 

Independent  review  team  officials  attributed  the  comments  not  being 
closed  to  the  fact  that  the  team  did  not  have  authority  over  the  cost  model 
to  ensure  that  its  comments  were  addressed.  Further,  these  officials  told 
us  that  they  were  not  asked  to  review  the  final  version  of  the  cost  model, 
which  was  the  version  that  first  introduced  alternative  3V,  and  their  review 
of  the  final  version  of  the  AOA  report  occurred  after  the  DON  CIO  had 
submitted  it  to  OSD  CAPE  for  final  approval. 

According  to  officials  responsible  for  developing  the  AOA,  the  weaknesses 
in  the  AOA  cost  estimates  largely  exist  because  there  was  not  enough  time 
to  conduct  a  more  thorough  analysis.  Specifically,  they  told  us  that  the 
AOA  schedule  was  constrained  because  the  program  wanted  to  get 
requests  for  proposals  for  NGEN  contracts  out  by  a  predetermined  date. 
This  position  was  also  supported  by  various  management  meeting  minutes 
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and  other  artifacts  that  we  reviewed.  However,  DOD  and  DON  officials 
disagreed  with  this  position  and  told  us  that  the  time  allotted  to  conduct 
the  AOA  did  not  negatively  impact  its  quality  or  scope.  A  time-constrained 
approach  is  not  consistent  with  DOD  guidance,  which  states  that  the  scope 
of  an  alternatives  analysis  should  be  proportionate  to  the  amount  of 
resources  affected  by  the  decision,  with  more  significant  programs 
receiving  more  analytical  attention.  The  combination  of  the  AOA 
weaknesses  we  identified,  and  the  fact  that  NGEN  has  a  preliminary 
program  life  cycle  cost  estimate  of  about  $50  billion  for  increment  1  and  is 
intended  to  provide  the  foundation  for  DON’s  future  networking 
environment,  suggest  the  need  for  considerable  analytical  attention  to 
alternative  approaches. 

Without  reliable  cost  estimates  for  each  alternative,  decision  makers  did 
not  have  a  sound  basis  for  making  an  informed  decision  on  an  NGEN 
solution.  Most  notably,  since  the  estimates  did  not  reflect  increased  risks 
associated  with  the  segmented  approach,  the  differences  in  the 
alternatives’  costs  were  understated,  and  the  amount  of  risk  and  costs 
accepted  by  proceeding  with  a  segmented  approach  were  not  fully 
understood. 


AOA  Did  Not  Sufficiently 
Assess  Operational 
Effectiveness 


While  the  AOA  identified  program  capabilities  and  goals,  it  did  not 
sufficiently  assess  the  alternatives’  operational  effectiveness,  making  it 
unclear  how  well  the  alternatives  would  actually  satisfy  NGEN  capabilities 
and  goals.  Specifically, 

•  The  AOA  identified  capabilities  and  goals  that  the  system  solution  should 
achieve.  Among  other  things,  these  included  addressing  NMCI  capability 
limitations  identified  based  on  8  years  of  operational  experience,  as  well 


In  addition  to  including  reliable  cost  estimates,  an  AOA  should  assess  how 
well  each  alternative  satisfies  required  capabilities  or  goals.  According  to 
DOD  guidance,14  such  an  analysis  should  (1)  identify  the  operational 
capabilities  and  goals  to  be  achieved  with  the  system  solution,  (2) 
establish  quantitative  or  qualitative  measures  for  evaluating  the 
operational  effectiveness  of  each  alternative,  and  (3)  assess  the  ability  of 
each  alternative  to  achieve  these  measures. 


14 Defense  Acquisition  Guidebook,  Section  3.3  “Analysis  of  Alternatives”  (Mar.  19,  2010); 
DON  Acquisition  and  Capabilities  Guidebook  (December  2008);  and  Air  Force  Analysis 
of  Alternatives  Handbook  (July  2008). 
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as  capabilities  needed  to  support  DOD  and  DON  networking  strategies  for 
DOD’s  Global  Information  Grid  Network  Operations  and  DON’s  future 
Naval  Networking  Environment.  (See  table  8  for  these  capabilities  and 
goals.) 

•  The  AOA  did  not  establish  quantitative  or  qualitative  measures  for 
assessing  the  alternatives’  ability  to  achieve  the  identified  NGEN 
capabilities  and  goals,  as  shown  in  table  8.  For  example,  one  of  the 
capabilities  was  visibility  into  root  causes  for  major  network  outages, 
which  the  AOA  merely  concluded  that  alternatives  2,  3V,  and  3  were 
equally  effective  in  addressing,  even  though  no  quantitative  or  qualitative 
measures  of  the  alternatives’  respective  ability  to  provide  visibility  into 
root  causes  were  defined.  Further,  the  AOA  did  not  discuss  the 
methodology  for  assessing  the  alternatives.  Rather,  it  simply  states  that  it 
was  a  qualitative  assessment. 

While  the  AOA  did  not  establish  measures  for  assessing  the  alternatives’ 
ability  to  achieve  NGEN  capabilities  and  goals,  it  did  establish  several 
quantitative  measures  to  differentiate  among  the  alternatives’  respective 
approaches  to  acquiring,  managing,  and  delivering  NMCI  capabilities. 
However,  these  measures  alone  do  not  provide  insight  into  how  they 
would  influence  the  operational  effectiveness  of  each  alternative  because 
they  were  not  linked  to  NGEN  capabilities  and  goals,  and  they  did  not 
provide  sufficient  insight  for  selecting  a  preferred  alternative.  For 
example,  while  the  AOA  recognized  that  an  increase  in  the  number  of 
contractual  relationships  would  result  in  more  complexity  and  risk  in 
implementing  the  alternative,  it  did  not  include  measures  for  quantifying 
how  much  more  risk  is  introduced  as  the  number  of  contractual 
relationships  increases.  (See  table  7  for  the  measures  that  were  provided 
in  the  AOA.) 
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Table  7:  Measures  Related  to  Overseeing,  Operating,  and  Managing  NGEN 

Status  quo 

Alternative  2 

Alternative  3V  Alternative  3 

Government  personnel 
as  a  percentage  of  total 
NGEN  personnel 

28 

39 

40 

41 

Number  of  seams3 
managed  by  the 
government 

1 

1 

36 

36 

Number  of  contractual 
relationships  managed 
by  the  government 

3 

3 

10 

15 

Source:  NGEN  Analysis  of  Alternatives  report,  Center  for  Naval  Analyses,  April  2009. 


“Seams  are  the  relationships  between  segments  (e.g.,  local  area  networks,  service  desks,  and 
enterprise  hardware). 

In  addition,  the  AOA  included  a  separate  assessment  of  the  likelihood  of 
each  alternative  to  successfully  implement  IT  best  practices  for  end-to-end 
IT  service  delivery  (i.e.,  IT  Service  Management  framework).15  To  DON’s 
credit,  the  approach  used  to  measure  the  alternatives  in  this  assessment 
was  more  structured  and  better  documented.  Specifically,  the  AOA  team 
conducted  table-top  exercises  with  subject-matter  experts  representing 
each  of  the  communities  that  will  contribute  to  the  acquisition,  operation, 
and  oversight  of  NGEN,  and  it  worked  through  scenarios,  such  as 
everyday  operations  and  responding  to  a  computer  network  incident,  to 
determine  the  extent  to  which  each  alternative  could  employ  IT  best 
practices  to  address  a  given  scenario.  The  team  captured  comments  made 
by  participants  and  used  them  to  infer  rankings  that  resulted  in  numerical 
scores  for  each  alternative. 

The  AOA  did  not  assess  the  alternatives’  ability  to  address  capabilities  and 
goals  using  defined  measures  of  operational  effectiveness  because,  as 
stated  previously,  no  measures  were  established.  Instead,  it  compared  the 
alternatives  based  on  qualitative  determinations  of  whether  the  capability 
or  goal  was  either  met  or  partially  met.  (See  table  8  for  the  results  of 
DON’s  assessment.) 


15To  support  DON’s  Naval  Networking  Environment  goals,  NGEN  plans  to  implement  an  IT 
service  management  framework  to  measure  and  report  on  all  aspects  of  end-to-end  IT 
service  delivery.  DON  plans  to  use  the  IT  Infrastructure  Library  best  practices  to 
implement  this  framework. 
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Table  8:  DON’S  Qualitative  Assessment  of  Alternatives’  Ability  to  Meet  NGEN  Capabilities  and  Goals 


Status  quo 

Alternative  2 

Alternative  3V 

Alternative  3 

Capabilities 

•  NMCI  capabilities  and  services  as  of  September  2010 

V 

✓ 

✓ 

•  Address  NMCI  deficiencies 

•  Solve  problem  with  out-of-scope  government  directed 
action 

✓- 

✓- 

✓ 

•  Sufficient  visibility/situational  awareness  of  network 
operations 

✓ 

✓ 

•  Visibility  into  root  causes 

✓ 

✓ 

✓ 

•  Adequate  log  keeping 

✓ 

V 

✓ 

•  Technology  refresh/architecture  upgrades 

✓ 

V 

✓ 

•  Network  Operations  Concept  of  Operations 

•  Support  Network  Operations  Concept  of  Operations 

✓ 

✓ 

•  Proactive  control/defense  of  network 

✓ 

✓ 

Goals 

•  Supports  Naval  Networking  Environment 

•  Enterprise  network  interoperability 

V 

V 

✓ 

•  Government  operational  control 

V 

✓ 

✓ 

•  Support  transformation  to  service-oriented  architecture 

V 

✓ 

✓ 

•  Open  architecture  and  standards 

V 

V 

✓ 

✓ 

•  Implement  IT  services  management 

V- 

✓- 

✓ 

•  Implement  portfolio  management  process 

V 

✓ 

✓ 

•  Active  monitor/report  of  service  level  agreements 

V 

V 

✓ 

✓ 

Source:  NGEN  Analysis  of  Alternatives  report,  Center  for  Naval  Analyses,  April  2009. 

Note:  A  checkmark  indicates  that  the  government's  goal  or  concern  would  be  addressed  by  the 
alternative;  a  check  minus  indicates  that  the  government’s  goal  or  concern  would  be  partially 
addressed  by  the  alternative. 

As  with  the  cost  estimates,  officials  responsible  for  developing  the  AOA 
told  us  that  the  operational  effectiveness  analysis  was  subject  to  time 
constraints  so  that  requests  for  proposals  could  be  issued  on  time. 
Although  DOD  and  DON  officials  told  us  that  the  time  allotted  to  conduct 
the  AOA  did  not  negatively  impact  its  quality  or  scope,  our  review  suggests 
otherwise.  Further,  the  time-constrained  approach  is  not  consistent  with 
DOD  guidance,  which  states  that  the  scope  of  an  alternatives  analysis 
should  be  proportionate  to  the  resources  affected  by  the  decision,  with 
more  significant  programs  receiving  more  analytical  attention. 
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Without  a  more  thorough  effectiveness  analysis,  decision  makers  did  not 
have  a  sound  basis  for  making  an  informed  decision  on  the  best  NGEN 
alternative  to  pursue.  Instead,  DON  has  selected  a  segmented  approach  on 
the  basis  that  it  would  provide  increased  flexibility  in  meeting  NGEN 
capabilities  and  goals  with  no  additional  cost,  even  though  the  degree  of 
increased  flexibility  among  the  alternatives  remains  unclear. 


Current  Acquisition 
Approach  Was  Not 
Addressed  in  the  AOA  and 
Is  Riskier  and  Costlier 
Than  Other  Approaches  in 
the  Analysis 


According  to  DOD  guidance, 16  an  AOA  should  examine  viable  solutions 
with  the  goal  of  identifying  the  most  promising  option,  thereby  informing 
acquisition  decision  making.  However,  the  segmented  approach  currently 
being  pursued  by  DON  was  not  one  of  the  alternatives  assessed  in  the 
AOA.  Specifically,  the  current  approach  has  more  contracts,  a  different 
segmentation  scheme,  and  a  different  transition  timeline  than  the  analyzed 
alternatives.  Further,  the  impact  of  these  differences  in  terms  of  how  they 
compare  to  the  original  alternatives  was  not  assessed. 


The  approach  that  is  being  pursued  by  the  program  office  includes  a 
higher  number  of  contracts  than  those  analyzed  in  the  AOA.  Given  that  the 
AOA  highlighted  greater  schedule  and  performance  risks  as  the  number  of 
contracts  and  contractual  relationships  in  the  approach  increase,  the 
relative  schedule  and  performance  risks  for  the  current  approach  are 
likely  greater  than  those  for  alternative  3,  and  therefore  are  likely  to  result 
in  greater  costs.  In  support  of  this  likelihood,  DON’s  November  2009  risk- 
adjusted  preliminary  program  life  cycle  cost  estimate  for  the  current 
approach  for  fiscal  year  2011  through  fiscal  year  2015  shows  that  the 
current  approach  will  cost  at  least  an  estimated  $4.7  billion  more  than  any 
of  the  alternatives  in  the  AOA.  (See  table  9  for  a  comparison  of  the  current 
approach  to  the  approaches  assessed  in  the  AOA  and  fig.  3  for  an 
illustration  of  the  contractual  relationships  associated  with  DON’s  current 
approach.) 


16 Defense  Acquisition  Guidebook,  Section  3.3  “Analysis  of  Alternatives”  (Mar.  19,  2010). 
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Table  9:  Comparison  of  NGEN  AOA  Alternatives  and  Current  Approach 


Status  quo 

Alternative  2 

Alternative  3V 

Alternative  3 

Current  approach 

Contracts3 

2 

2 

4 

5 

6 

Contractual  relationships 

3 

3 

10 

15 

21 

Fiscal  year  2015 
operating  cost  (billions) 

$2.25 

$2.62 

$2.64 

$2.65 

$2.94 

Fiscal  years  2011  to  2015 
total  cost  (billions) 

$10.3 

$10.8 

$10.8 

$10.7 

$15.6 

Relative  schedule  risk 

Least 

More 

Greater 

Greatest 

(not  determined) 

Relative  risk  in  delivering 
planned  performance 

Least 

More 

Greater 

Greatest 

(not  determined) 

Transition  timeline 

24  months  of 
continued  NMCI 
services  before  24 
months  of 
transitioning  to  new 
provider,  and  then 

12  months  of 

NGEN  steady-state 

Same  timeline  as 
alternative  1 

Same  timeline  as 
alternative  1 

Same  timeline  as 
alternative  1 

14  months  of 
continued  NMCI 
services  before  29 
months  of  transitioning 
to  new  providers  in 
staggered  phases, 
and  then  17  months  of 
NGEN  steady-state 

Sources:  NGEN  Analysis  of  Alternatives  report,  Center  for  Naval  Analyses,  April  2009  (status  quo  and  alternatives  2,  3V,  and  3);  GAO 
analysis  of  DON  data  (current  approach). 


Note:  The  difference  between  the  total  fiscal  year  2011  to  fiscal  year  2015  costforthe  current 
approach  and  alternative  3V  is  about  $4.7  billion  due  to  rounding. 

aln  all  alternatives,  wide  area  network  services  will  be  provided  by  the  Defense  Information  Systems 
Agency.  DON'S  relationship  with  the  Defense  Information  Systems  Agency  will  be  contract-like  in  that 
it  is  an  interagency  agreement,  in  which  DON  must  pay  forservices  and  will  have  to  manage  its 
relationship  with  the  service  providers. 
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Figure  3:  Contractual  Relationships  in  Current  NGEN  Approach 


OSD  CAPE  officials  told  us  that  they  believe  the  differences  between  the 
current  approach  and  alternatives  assessed  in  the  AOA  are  not  significant 
because  DON  is  still  pursing  a  segmented  approach  and  that  the 
differences  were  the  result  of  “an  appropriate  evolution  of  the  segmented 
approach.”  They  further  said  that  the  increased  risks  in  the  current 
approach  are  offset  by  mitigating  factors,  such  as  the  use  of  staggered 
phases  to  implement  NGEN  and  the  use  of  more  efficient  segmentation 
schemes.  However,  we  have  yet  to  receive  any  analysis  to  support  their 
positions,  and  the  current  approach  is  estimated  to  cost  about  $4.7  billion 
more.  As  a  result,  DON  cannot  demonstrate  that  it  is  pursuing  the  most 
cost-effective  approach  for  acquiring  NGEN  capabilities  and  meeting 
NGEN  goals. 


NGEN’s  Schedule 
Does  Not  Provide  a 
Reliable  Basis  for 
Program  Execution, 
and  Program  Delays 
Are  Occurring 


The  success  of  a  large-scale  acquisition  program  depends  in  part  on  having 
a  reliable  schedule  that  defines,  among  other  things,  when  work  activities 
and  milestone  events  will  occur,  how  long  they  will  take,  and  how  they  are 
related  to  one  another.  As  such,  the  schedule  not  only  provides  a  road  map 
for  systematic  program  execution  but  also  provides  the  means  by  which  to 
gauge  progress,  identify  and  address  potential  problems,  and  promote 
accountability.  Without  a  reliable  schedule,  it  is  likely  that  established 
program  milestones  will  slip.  In  the  case  of  NGEN,  such  delays  are  already 
being  experienced. 
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Our  work  has  identified  nine  best  practices  associated  with  developing 
and  maintaining  a  reliable  schedule.  These  are  (1)  capturing  all  activities, 
(2)  sequencing  all  activities,  (3)  assigning  resources  to  all  activities,  (4) 
establishing  the  duration  of  all  activities,  (5)  integrating  schedule  activities 
horizontally  and  vertically,  (6)  establishing  the  critical  path17  for  all 
activities,  (7)  identifying  reasonable  “float”18  between  activities,  (8) 
conducting  a  schedule  risk  analysis,  and  (9)  updating  the  schedule  using 
logic  and  durations.  See  table  10  for  a  description  of  each  of  these  best 
practices. 


Table  10:  Description  of  Scheduling  Best  Practices 

Practice 

Description 

Capturing  all  activities 

The  schedule  should  reflect  all  activities  (steps,  events,  outcomes,  etc.)  as  defined  in 
the  program's  work  breakdown  structure  to  include  activities  to  be  performed  by  both 
the  government  and  its  contractors. 

Sequencing  all  activities 

The  schedule  should  sequence  activities  in  the  order  that  they  are  to  be  implemented. 

In  particular,  activities  that  must  finish  prior  to  the  start  of  other  activities  (i.e., 
predecessor  activities),  as  well  as  activities  that  cannot  begin  until  other  activities  are 
completed  (i.e.,  successor  activities)  should  be  identified. 

Assigning  resources  to  all  activities 

The  schedule  should  reflect  who  will  do  the  work  activities,  whether  all  required 
resources  will  be  available  when  they  are  needed,  and  whether  any  funding  or  time 
constraints  exist. 

Establishing  the  duration  of  all  activities 

The  schedule  should  reflect  the  duration  of  each  activity.  These  durations  should  be  as 
short  as  possible  and  have  specific  start  and  end  dates. 

Integrating  schedule  activities  horizontally 
and  vertically 

The  schedule  should  be  horizontally  integrated,  meaning  that  it  should  link  the 
products  and  outcomes  associated  with  sequenced  activities.  The  schedule  should 
also  be  vertically  integrated,  meaning  that  traceability  exists  among  varying  levels  of 
activities  and  supporting  tasks  and  subtasks. 

Establishing  the  critical  path  for  all  activities 

The  critical  path  represents  the  chain  of  dependent  activities  with  the  longest  total 
duration  in  the  schedule. 

Identifying  reasonable  float  between  activities 

The  schedule  should  identify  a  reasonable  amount  of  float—  the  time  that  a 
predecessor  activity  can  slip  before  the  delay  affects  successor  activities—  so  that 
schedule  flexibility  can  be  determined.  As  a  general  rule,  activities  along  the  critical 
path  typically  have  the  least  amount  of  float. 

11  The  critical  path  represents  the  chain  of  dependent  activities  with  the  longest  total 
duration  in  the  schedule. 

lsFloat  is  the  amount  of  time  a  task  can  slip  before  affecting  the  critical  path. 
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Practice 

Description 

Conducting  a  schedule  risk  analysis 

A  schedule  risk  analysis  is  used  to  predict  the  level  of  confidence  in  the  schedule, 
determine  the  amount  of  time  contingency  needed,  and  identify  high-priority  schedule 
risks. 

Updating  the  schedule  using  logic  and 
durations 

The  schedule  should  use  logic  and  durations  in  order  to  reflect  realistic  start  and 
completion  dates,  be  continually  monitored  to  determine  differences  between 
forecasted  completion  dates  and  planned  dates,  and  avoid  logic  overrides  and  artificial 
constraint  dates. 

Source:  GAO. 

In  December  2009,  NGEN  established  a  baseline  integrated  master 
schedule19  composed  of  over  25  separate  underlying  schedules  (or 
subschedules)  to  capture  program  milestones  and  the  expected 
completion  dates  for  activities  leading  up  to  them.  However,  the  most 
current  version  of  this  schedule  (May  2010)  that  was  available  at  the  time 
we  began  our  review  was  not  reliable  because  only  two  of  the  four 
subschedules20  that  we  analyzed  substantially  met  any  of  the  nine 
practices.  The  results  of  our  analysis  of  the  four  subschedules  are 
summarized  in  table  11. 

19This  schedule,  considered  to  be  an  informal  baseline  schedule,  was  approved  by  the 
program  manager  to  allow  for  later  comparison  to  the  current  plan  and  to  manage  the 
overall  scope  of  work  to  be  conducted  within  the  program.  An  official  baseline  schedule  is 
expected  to  be  approved  by  the  USD  (AT&L)  as  part  of  the  Acquisition  Program  Baseline  at 
Milestone  C. 

20These  schedules  were  the  Transition  Integrated  Product  Team,  IT  Service  Management 
Process  Development,  Comprehensive  Facilities  Inventory  Phase  2  Plan  of  Action  and 
Milestones,  and  Contract  Technical  Representative  Workforce  Reconstitution.  These 
subschedules  represented  the  ongoing  and/or  planned  work  on  NGEN  to  be  completed 
prior  to  the  deployment  phase,  as  well  as  high-level  plans  for  remaining  work  to  be 
completed  postdeployment. 


Page  23 


GAO-11-150  Information  Technology 


Table  1 1 :  Summary  Results  of  Selected  NGEN  Schedules’  Satisfaction  of  Schedule  Estimating  Practices 


Schedule  practice 

Transition 
Integrated 
Product  Team 

IT  Service 
Management 
Process 
Development 

Comprehensive 
Facilities  Inventory 
Phase  2  Plan  of  Action 
and  Milestones 

Contract  Technical 
Representative 
Workforce 
Reconstitution 

Capturing  all  activities 

1 

© 

© 

© 

Sequencing  all  activities 

© 

© 

© 

© 

Assigning  resources  to  all  activities 

O 

o 

O 

O 

Establishing  the  duration  of  all  activities 

© 

• 

© 

• 

Integrating  schedule  activities  horizontally 
and  vertically 

© 

© 

© 

© 

Establishing  the  critical  path  for  all 
activities 

© 

© 

© 

© 

Identifying  reasonable  float  between 
activities 

© 

© 

© 

© 

Conducting  a  schedule  risk  analysis 

o 

o 

o 

o 

Updating  the  schedule  using  logic  and 
durations 

© 

© 

© 

© 

Source:  GAO  analysis  of  DON  data. 

•  "Met"  =  DON  provided  complete  evidence  that  satisfied  the  criterion. 

®  "Substantially  Met”  =  DON  provided  evidence  thatsatisfied  more  than  one-half  of  the  criterion. 

®  "Partially  Met”  =  DON  provided  evidence  thatsatisfied  about  one-half  of  the  criterion. 

®  "Minimally  Met”  =  DON  provided  evidence  thatsatisfied  less  than  one-half  of  the  criterion. 

O  "Not  Met”  =  DON  provided  no  evidence  thatsatisfied  any  portion  of  the  criterion. 

Note:  The  Transition  Integrated  Product  Team  subschedule  contains  work  associated  with  transition 
planning,  such  as  development  of  the  T ransition  Management  Strategy  and  T ransition  Concept  of 
Operations.  The  IT  Service  ManagementProcess  Developmentsubschedule  contains  early  transition 
activity  work  focused  on  developing  service  management  processes  and  procedures.  The 
Comprehensive  Facilities  Inventory  Phase  2  Plan  of  Action  and  Milestones  subschedule  contains 
early  transition  activity  work  associated  the  evaluation,  analysis,  and  validation  of  the  current  NMCI 
infrastructure  inventory.  The  ContractTechnical  Representative  Workforce  Reconstitution 
subschedule  contains  early  transition  activity  work  associated  with  job  task  analysis  and  the 
assessmentof  learning  tools  for  contract  technical  representatives. 

•  Capturing  all  activities.  All  four  subschedules  partially  met  this  practice. 
Specifically,  the  majority  of  the  activities  contained  in  these  subschedules 
could  be  mapped  back  to  the  program’s  NGEN  work  breakdown  structure. 
However,  this  structure  is  defined  at  a  high  level  and  is  not  expected  to  be 
further  decomposed  into  planned  work  products  and  deliverables  until  the 
program  enters  the  deployment  phase  when  NGEN  contracts  are  awarded. 
Until  this  structure  is  sufficiently  defined,  it  cannot  be  determined  whether 
the  program  schedules  capture  all  work  needed  to  accomplish  program 
objectives.  For  example,  we  identified  risk  mitigation  activities  for  10 
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active  risks  that  should  have  been,  but  were  not,  captured  as  scheduled 
work. 

During  our  review,  program  officials  told  us  that  they  had  since  taken 
steps  to  ensure  that  all  risk  mitigation  activities  are  added  to  the  schedule. 
However,  until  NGEN  work  is  sufficiently  defined,  the  program  does  not 
have  complete  assurance  that  the  activities  currently  captured  in  the 
various  schedules  support  NGEN  increment  1. 

•  Sequencing  all  activities.  One  subschedule  substantially  met  this  practice 
while  the  other  three  minimally  met  it.  The  subschedule  that  substantially 
met  this  practice  had  less  than  1  percent  of  activities  missing  a 
predecessor  or  successor  dependency.  Of  the  remaining  three 
subschedules,  two  did  not  identify  predecessor  or  successor  activities  for 
over  half  of  the  activities  in  their  schedules.  This  is  of  concern  because  if 
an  activity  that  has  no  logical  successor  slips,  the  schedule  will  not  reflect 
the  effect  of  these  slips  on  the  critical  path,  float,  or  scheduled  start  dates 
of  “downstream”  (i.e.,  later)  activities.  Additionally,  one  subschedule  had 
“constraints”  placed  on  about  73  percent  of  its  activities,  meaning  that 
these  activities  cannot  begin  earlier  even  if  upstream  work  is  completed 
ahead  of  schedule.  According  to  program  officials,  they  are  working  to 
reduce  the  number  of  constraints  in  the  schedule.  However,  until  activities 
are  properly  sequenced,  these  issues  reduce  the  credibility  of  the  dates 
calculated  by  the  scheduling  tool. 

•  Assigning  resources  to  all  activities.  Program  officials  told  us  that  they 
do  not  assign  resources  to  any  of  the  program  schedules.  They  stated  that 
the  effort  necessary  to  assign  resources  within  the  schedules  would  be 
significant  and  that  they  did  not  have  the  staff  available  to  do  this. 
However,  without  proper  allocation  of  resources  in  the  schedule,  the 
program  office  cannot  accurately  forecast  the  likelihood  that  activities  will 
be  completed  by  their  projected  end  dates,  and  the  risk  that  key 
milestones  will  slip  increases. 

•  Establishing  the  duration  of  all  activities.  Two  subschedules  met  this 
practice  while  two  only  minimally  met  it.  The  two  subschedules  that  met 
this  practice  had  established  activities  with  reasonable  durations — the 
majority  of  which  were  under  30  days.  The  remaining  two  did  not  establish 
reasonable  durations  for  their  activities.  For  example,  the  majority  of  the 
activities  that  were  in  progress  for  the  Transition  Integrated  Product  Team 
subschedule  had  durations  ranging  from  50  days  to  1000  days.  When  such 
long  durations  are  assigned  to  work  activities,  it  is  likely  that  the  activity  is 
not  defined  to  the  necessary  level  to  identify  all  the  work  that  must  be 
performed. 
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•  Integrating  schedule  activities  vertically  and  horizontally.  One  of  the 
subschedules  substantially  met  and  the  other  three  partially  met  this 
practice.  The  subschedule  that  substantially  met  the  practice  is 
horizontally  aligned,  meaning  activities  are  logically  sequenced,  and 
vertically  aligned,  meaning  that  detailed  activities  roll  up  into  larger 
summary  tasks.  The  other  three  subschedules  are  also  vertically  aligned; 
however,  they  are  unable  to  demonstrate  horizontal  integration  because, 
as  previously  discussed,  activities  were  not  all  logically  sequenced. 

The  integration  issues  identified  on  these  subschedules  also  impact  the 
NGEN  master  schedule.  Because  of  the  high  number  of  missing 
dependencies,  the  number  of  in-progress  activities  with  durations 
exceeding  30  days,  and  the  high  number  of  constraints,  the  master 
schedule  is  likely  not  fully  horizontally  integrated. 

Further,  one  of  the  subschedules  is  not  vertically  aligned  with  the  master 
schedule  because  none  of  the  key  work  activities  in  the  subschedule  were 
included  in  the  master  schedule.  In  addition,  the  master  schedule  was  not 
integrated  with  the  approved  NGEN  acquisition  strategy.  Program  officials 
told  us  they  did  not  revise  the  dates  in  the  master  schedule  until  after  the 
continuity  of  services  contract  was  awarded  (July  2010),  and  that  the  dates 
in  the  acquisition  strategy  reflected  the  current  information.  By  using  a 
source  other  than  the  program  office’s  working  schedule,  oversight 
officials’  expectations  about  when  milestones  will  be  met  may  not  be 
realistic. 

•  Establishing  the  critical  path  for  all  activities.  None  of  the  four 
subschedules  fully  met  this  practice.  Specifically,  the  scheduling  tool  was 
unable  to  generate  a  valid  critical  path  for  the  subschedules  due  to  the 
extent  of  issues  associated  with  the  sequencing  of  activities,  integration  of 
activities,  and  identification  of  reasonable  float  (discussed  below). 
Program  officials  stated  that  they  do  not  manage  a  critical  path  generated 
by  the  scheduling  tool.  Instead,  these  officials  stated  that  they  track 
activities  associated  with  the  deployment  phase  decision  (Milestone  C), 
which  they  have  designated  as  being  critically  important  to  them. 

However,  such  practice  does  not  allow  the  program  to  have  immediate 
insight  into  the  full  sequence  of  activities  (both  critical  and  noncritical) 
that,  if  delayed,  would  impact  the  planned  completion  date  of  Milestone  C, 
as  well  as  a  projected  completion  date  should  one  of  these  activities  be 
delayed. 

•  Identifying  reasonable  float  between  activities.  Two  subschedules 
partially  met  this  practice,  while  the  remaining  two  minimally  met  it.  Each 
of  these  subschedules  identified  float;  however,  the  amount  of  excessive 
float  varied.  Both  the  Contract  Technical  Representative  Workforce 
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Reconstitution  and  IT  Service  Management  Process  Development 
subschedules  partially  met  this  practice  because  only  25  percent  and  41 
percent  of  their  work  activities  had  float  of  100  days  or  greater, 
respectively.  The  two  remaining  subschedules  minimally  met  this  practice 
because  over  60  percent  of  their  activities  contained  float  of  100  days  or 
greater. 

Excessive  float  values  are  indicative  of  schedule  logic  that  is  flawed, 
broken,  or  absent.  As  such,  these  float  values  are  of  limited  value  to 
mitigate  risk  by  reallocating  resources  from  tasks  that  can  safely  slip  to 
tasks  that  must  be  completed  on  time. 

•  Conducting  a  schedule  risk  analysis.  The  program  has  not  performed  a 
schedule  risk  analysis.  Instead,  according  to  program  officials,  schedule 
risks  are  considered  during  risk  management  board  meetings  and  program 
health  assessments.  However,  without  this  analysis,  it  is  not  possible  to 
determine  a  level  of  confidence  in  meeting  program  milestones.  A 
schedule  risk  analysis  will  calculate  schedule  reserve,  which  can  be  set 
aside  for  those  activities  identified  as  high-risk.  Without  this  reserve,  the 
program  faces  the  risk  of  delays  if  they  were  to  occur  on  critical  path 
activities. 

•  Updating  the  schedule  using  logic  and  durations.  All  four  subschedules 
partially  met  this  practice.  According  to  program  officials,  status  updates 
are  performed  on  the  subschedules  once  a  week.  However,  despite  status 
updates,  date  anomalies  exist.  For  example,  the  Contract  Technical 
Representative  Workforce  Reconstitution  subschedule  included  five 
activities  with  an  actual  start  date  in  the  future.  Furthermore,  the 
subschedules’  inability  to  produce  a  valid  critical  path  indicates  that  the 
sequencing  of  activities  is  not  appropriate,  thus  impairing  the  scheduling 
tool’s  ability  to  generate  realistic  start  and  end  dates. 

According  to  program  officials,  they  were  aware  of  some  of  these  schedule 
weaknesses  based  on  a  May  2010  assessment  of  the  schedule  performed 
by  a  support  contractor.  Among  other  things,  the  contractor’s  assessment 
found  that  the  schedule  did  not  provide  for  stakeholder  review  of  most  of 
the  major  acquisition  documents  or  steps  to  mitigate  known  risks,  and  that 
it  lacked  a  valid  critical  path  due  to  network  logic  issues  and  activity 
constraints.  Officials  told  us  that  they  plan  to  address  these  issues. 

In  addition,  program  officials  stated  that  they  hold  monthly  program 
management  reviews  to  discuss  schedule  quality  issues,  as  well  as  risks  or 
issues  that  might  affect  the  schedule.  However,  these  reviews  are  not 
addressing  key  schedule  issues.  Specifically,  the  NGEN  schedule 
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management  plan  calls  for  the  schedule  to  be  resource-loaded  from  a 
centralized  resource  pool  approved  by  the  program  manager,  activities 
beginning  within  90  days  to  have  durations  of  no  more  than  20  days,  and 
activities  for  mitigating  approved  program  risks  to  be  added  to  the 
schedule.  However,  our  analysis  of  the  schedule  showed  that 

•  resources  are  not  assigned  within  the  schedule, 

•  activities  that  are  to  begin  within  90  days  have  durations  that  exceed  20 
days,  and 

•  activities  for  mitigating  10  approved  program  risks  were  not  included. 

Collectively,  the  weaknesses  in  implementing  the  nine  key  practices  for 
the  program’s  integrated  master  schedule  increase  the  risk  of  schedule 
slippages  and  related  cost  overruns  and  make  meaningful  measurement 
and  oversight  of  program  status  and  progress,  as  well  as  accountability  for 
results,  difficult  to  achieve.  Moreover,  they  undermine  the  schedule’s 
ability  to  produce  credible  dates  for  planned  NGEN  milestones  and  events. 
In  the  case  of  increment  1,  this  risk  has  already  been  realized.  Specifically, 
the  NGEN  master  schedule  was  rebaselined  in  August  2010,  resulting  in 
delays  in  a  number  of  key  dates,  including  a  5-month  delay  of  the 
Milestone  C  decision.  See  table  12  for  a  summary  of  key  event  and 
milestone  delays. 


Table  12:  Delays  in  Key  NGEN  Program  Events  and  Milestones,  as  of  August  2010 


Event/milestone 

Status  of 
event/milestone 

Delay  in 
months’ 

Milestone  decisions 

Materiel  Development  Decision 

Completed 

3 

DON  Gate  5  Review 

In  progress 

4 

DON  Gate  6  Review 

Notyet  occurred 

7 

Milestone  C 

Not  yet  occurred 

5 

Documents  complete 

Acquisition  Strategy 

Completed 

4 

Economic  Analysis 

In  progress 

0 

Cost  Analysis  Requirements  Description 

In  progress 

4 

Preliminary  Program  Life  Cycle  Cost  Estimate 

In  progress 

4 

Capability  Production  Document 

In  progress 

9 

Systems  Engineering  Plan 

In  progress 

10 
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Event/milestone 

Status  of 
event/milestone 

Delay  in 
months3 

Milestone  decisions 

Testand  Evaluation  Master  Plan 

In  progress 

14 

Acquisition  Program  Baseline 

In  progress 

3 

Contracts 

Independent  Security  Operations,  Oversightand 
Assessment  Support  Contract  Award 

Not  yet  awarded 

4 

Continuity  of  Services  Contract  Award 

Completed 

5 

Transport  Requestfor  Proposal  Release 

Not  yet  released 

4 

Transport  Contract  Award 

Notyet  awarded 

5 

Hardware  Contract  Award 

Notyet  awarded 

0 

Software  Contract  Award 

Not  yet  awarded 

0 

Enterprise  Services  Requestfor  Proposal 

Release 

Not  yet  released 

0 

Enterprise  Services  Contract  Award 

Not  yet  awarded 

2 

Source:  GAO  analysis  of  DON  data. 


"Delays  were  calculated  by  comparing  data  in  the  December  2009  schedule  with  the  data  in  the 
August  2010  schedule.  Delays  were  rounded  to  the  nearest  month. 

While  officials  stated  that  they  have  addressed  some  of  the  weaknesses 
identified  above  in  the  August  2010  rebaselined  integrated  master 
schedule,  they  conceded  that  this  schedule  does  not  assign  resources  to 
work  activities,  and  the  scheduling  tool  is  unable  to  generate  a  valid 
critical  path.  Because  these  key  scheduling  practices  are  not  being 
performed,  the  schedule  is  still  not  reliable. 

Without  a  fully  integrated  and  reliably  derived  schedule  for  the  entire 
NGEN  program,  the  program  office  cannot  identify  when  and  how  it  will 
proceed  through  Milestone  C  and  ultimately  transition  from  NMCI  to 
NGEN,  and  it  cannot  adequately  manage  and  measure  its  progress  in 
executing  the  work  needed  to  do  so. 
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NGEN  Acquisition 
Decisions  Were  Not 
Always  Performance- 
and  Risk-Based 


Successful  execution  of  system  acquisition  programs  depends  in  part  on 
effective  executive-level  governance,  to  include  having  organizational 
executives  review  these  programs  at  key  milestones  in  their  life  cycles  and 
make  informed  performance-  and  risk-based  decisions  as  to  how  they 
should  proceed.21  DON  policy  recognizes  the  importance  of  such  milestone 
reviews.  According  to  this  policy,  acquisition  programs  must  proceed 
through  a  series  of  gate  reviews  (as  discussed  above),  during  which 
program  performance  is  assessed  and  satisfactory  program  health  must  be 
demonstrated  prior  to  moving  forward. 


Currently,  program  performance  and  health  at  each  gate  are  assessed 
using  the  Naval  Probability  of  Program  Success  assessment  methodology, 
which  was  established  in  September  2008.  This  assessment  addresses  four 
aspects  of  a  program:  (1)  requirements,  (2)  resources,  (3) 
planning/execution,  and  (4)  external  influencers.  Associated  with  each 
aspect  are  two  or  more  metrics,  each  of  which  is  scored  based  on 
underlying  criteria  that  are  unique  to  each  gate.  (See  table  13  for  a 
description  of  each  metric.)  At  a  given  gate  review,  the  criteria  are  rated  in 
terms  of  green,  yellow,  or  red.22  Further,  the  metrics  can  be  designated  as 
critical,  meaning  that  any  significant  issues  that  are  associated  with  these 
metrics  must  be  resolved  before  the  gate  can  be  exited. 


21GAO,  Information  Technology:  Federal  Agencies  Need  to  Strengthen  Investment  Board 
Oversight  of  Poorly  Planned  and  Performing  Projects,  GAO-09-566  (Washington,  D.C.: 
June  30,  2009). 

22Criteria  are  rated  green,  yellow,  or  red  based  on  the  extent  to  which  the  program  meets 
necessary  elements  associated  with  that  criteria.  These  measures  can  vary  and  are  defined 
specifically  for  each  criterion.  For  example,  it  could  be  based  on  the  status  of  a  deliverable 
(i.e.,  green  if  completed,  yellow  if  in  progress,  or  red  if  not  yet  started),  the  significance  of 
issues  (i.e.,  green  if  no  issues,  yellow  if  some  issues,  or  red  if  significant  issues),  a 
percentage  (i.e.,  green  if  less  than  10  percent,  yellow  if  between  10  percent  and  30  percent, 
or  red  if  greater  than  30  percent  cost  growth),  or  schedule  (i.e.,  green  if  less  than  6  months, 
yellow  if  between  6  and  12  months,  or  red  if  greater  than  12  months  schedule  slip). 
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Table  13:  Naval  Probability  of  Program  Success  Metric  Definitions 


Program  aspect 

Metric 

Definition 

Requirements 

Parameter  status 

Progress  toward  defining  capability  requirements  and  meeting  those  requirements; 
requirements  traceability  and  validity  of  the  threat  assessment. 

Scope  evolution 

Stability  of  performance  parameters  from  the  established  baseline  and  the  impact 
of  requirements  changes  on  total  program  costand  schedule. 

Concept  of  operations 

Progress  toward  developing  and  scoping  the  concept  of  operations,  using  it  to 
inform  program  requirements,  acquisition  approaches,  and  strategies,  and  the 
validity  of  the  concept  of  operations  over  time. 

Resources 

Budgetand  planning 

Sufficiency  of  funding  based  on  last  approved  budget  controls  and  degree  of 
deviation  from  the  current  cost  estimate.  Also  assesses  how  well  acquisition, 
systems  development,  and  sustainment  strategies  are  evolving  in  ways  intended  to 
mitigate  Total  Ownership  Cost  growth. 

Manning 

Stability  and  adequacy  of  Resource  Sponsor  and  Program  Office  staffing 
(availability,  skills,  experience,  certification,  and  training). 

Planning/execution 

Acquisition  management 

Completeness  of  the  program  master  schedule,  status  of  milestone  documentation 
development,  and  progress  toward  defining  derived  requirements  in  the  System 
Design  Specification. 

Industry/company 

assessment 

Assesses  market  research  activities,  industrial  base  health,  and  an  understanding 
of  industrial  implications  for  cost,  schedule,  and  technical  risks.  For  major  contracts, 
assesses  each  company's  financial  health,  financial  systems,  and  manufacturing/ 
production  capabilities. 

Total  ownership  cost 
estimating 

Adequacy  of  the  elements  required  to  produce  sound  cost  estimates:  program 
description  information,  cost  data,  cost  estimating  process,  cost  estimate  stability 
and  comparisons,  and  cost  estimate  measures. 

Test  and  evaluation 

Progress  toward  defining  and  executing  the  Test  and  Evaluation  Strategy  and  Test 
and  Evaluation  Master  Plan. 

Technical  maturity 

Maturity  of  the  system  and  subsystems  design,  as  well  as  the  technical  maturity  of 
critical  technology  elements  in  accordance  with  an  approved  Technology 
Development  Strategy. 

Sustainment 

Progress  toward  defining  and  executing  the  sustainment  strategy,  and  the  resource 
adequacy  applied  toward  the  life  cycle  sustainment  activities. 

Software 

Status  of  software  management  and  engineering  activities  by  government  agencies 
and/or  contractors  that  are  integral  to  program  deliverables. 

Contract  planning  and 
execution 

Status  of  procurement  activities  and  achievement  of  contracting  milestones  against 
the  planned  schedule;  performance  of  major  contractors  and/or  government 
performers. 

Government  program 
office  performance 

Progress  toward  defining  and  executing  intragovernment  requirements; 
responsiveness  to  deliverable  submissions;  delivery  of  facilities,  funding,  and 
government  furnished  equipment/information  in  accordance  with  scheduled 
requirements;  and  effectiveness  of  configuration  management  and  risk 
management  boards. 

Technology  protection3 

Progress  of  safeguarding  the  research,  technology  information,  and  applied 
knowledge  associated  with  the  program. 

External  influencers 

Fit  in  vision 

Alignment  with  current  documented  OSD  guidance  and  Navy/Marine  Corps 
strategies. 
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Program  aspect  Metric 

Definition 

Program  advocacy 

Support  from  key  stakeholders,  such  as  USD  (AT&L)  (or  equivalent),  CAPE, 
international  partners,  and  other  military  services. 

Interdependencies 

Status  of  dependent  programs'  delivery  of  the  requisite  capability  or  quantity  on 
schedule. 

Source:  DON  Naval  Probability  of  Program  Success  Guidebook  version  2.2. 


"This  was  a  new  metric  introduced  in  the  Naval  Probability  of  Program  Success  version  2  framework. 

As  noted  earlier,  a  Gate  1  review  was  not  held  because  the  gate-review 
process  was  not  established  when  the  program  began.  In  lieu  of  a  Gate  1 
review,  according  to  the  NGEN  Acquisition  Strategy,  the  Chief  of  Naval 
Operations  Executive  Board  met  to  confirm  NGEN  requirements  during 
the  winter  of  2007/2008  and  these  meetings  were  “nominally”  a  Gate  1 
review.  Subsequent  to  the  establishment  of  the  DON  gate  process,  an 
NGEN  Gate  2  review — intended  to  focus  on  an  analysis  of  alternatives — 
was  waived  in  early  2008  because  the  department  planned  to  continue  the 
use  of  existing  NMCI  technology,  and  NGEN  entered  the  DON  review 
process  at  Gate  3  in  April  2008.  OSD  later  identified  the  program  as  a  pre- 
MAIS  acquisition,  resulting  in  the  direction  to  conduct  an  analysis  of 
acquisition  alternative  approaches.  As  such,  DON  held  a  Gate  2  review  in 
April  2009,  one  year  after  the  Gate  3  review.  Since  then,  DON  held  a  Gate  4 
review  in  November  2009,  as  well  as  a  Gate  5  review  in  October  2010.  As 
discussed  below,  the  extent  to  which  each  of  the  gate  reviews  was 
performance-  and  risk-based  varied. 

•  Gate  3  review.  At  the  time  of  this  review,  which  was  in  April  2008,  the 
Probability  of  Program  Success  assessment  methodology  was  not  yet  in 
place.  Therefore,  program  review  documentation  focused  on,  for  example, 
program  activities  that  had  been  completed,  were  under  way,  and  were 
planned.  However,  these  activities  were  not  presented  relative  to  any 
benchmark  or  goal,  and  thus  program  performance  was  not  apparent  in 
the  documentation.  Further,  while  program  documentation  shows  that 
risks  were  disclosed,  such  as  funding  shortfalls  for  fiscal  years  2008  and 
2009,  as  well  as  workforce  and  training  challenges,  the  scope  and  nature  of 
the  information  presented  did  not  extend  to  the  level  that  the  assessment 
methodology  provides.  For  example,  the  information  presented  did  not 
address  the  realism  and  achievability  of  the  program  master  schedule  and 
the  confidence  level  associated  with  the  current  cost  estimate,  including 
the  difference  between  the  program  office  and  independent  cost  estimates, 
which  are  both  relevant  criteria  under  the  assessment  methodology  for  the 
gate.  Notwithstanding  these  gaps  in  information  that  would  have  limited 
informed  program  decision  making,  the  program  was  approved  to 
proceed. 
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•  Gate  2  review.  At  the  time  of  this  review,  which  was  in  April  2009,  the 
Probability  of  Program  Success  assessment  methodology  was  in  place. 
However,  it  was  not  used  to  inform  program  decision  making.  Instead,  the 
review  focused  on  the  AOA,  next  steps,  and  the  overall  program  timeline. 
While  briefing  documentation  shows  that  cost  estimates  for  the 
alternatives  exceeded  planned  funding,  the  documentation  did  not 
disclose  the  range  of  AOA  and  integrated  master  schedule  weaknesses 
discussed  earlier  in  this  report,  and  the  risks  associated  with  these 
limitations.  This  is  significant  because  the  Gate  2  assessment  criteria  focus 
on,  among  other  things,  whether  the  AOA  cost  estimates  and  master 
program  schedule  are  reliable  and  whether  program  execution  is  on 
schedule.  Notwithstanding  these  weaknesses,  the  program  was  approved 
to  proceed. 

•  Gate  4  review.  For  this  review,  DON  used  its  Probability  of  Program 
Success  methodology  and  assessed  the  health  of  the  program  against  each 
of  the  17  metrics,23  including  3  that  DON  designated  as  potentially 
critical — parameter  status,  budget  and  planning,  and  acquisition 
management.  According  to  the  program  health  assessment  used  at  this 
gate,  8  of  the  17  metrics  were  rated  as  red,  meaning  that  the  program  had 
identified  significant  issues  that  would  inhibit  delivery  of  capability  within 
approved  cost  and  schedule  constraints  and  that  mitigation  strategies  had 
not  been  identified.  Moreover,  the  8  metrics  rated  as  red  included  3  that 
were  designated  as  critical,  meaning  that  these  issues  needed  to  be 
resolved  before  exiting  the  gate.  Specifically,  the  parameter  status  metric 
was  rated  as  red  because  NGEN  requirements  that  increment  1  is  to  meet 
had  not  yet  been  defined;  the  budget  and  planning  metric  was  rated  as  red 
because  the  program  was  not  fully  funded;  and  the  acquisition 
management  metric  was  rated  as  red  because  the  USD  (AT&L)  had  yet  to 
authorize  the  milestone  at  which  the  program  would  enter  the  Defense 
Acquisition  System.  (See  fig.  4  for  the  assessment  results  for  all  17 
metrics.)  Moreover,  the  gate  briefing  document  highlighted  a  number  of 
risks  facing  the  program.  For  example,  it  faced  the  risk  that  key  program 
documentation,  such  as  the  System  Engineering  Plan  and  the  Test  and 
Evaluation  Master  Plan,  would  not  be  completed  until  NGEN  requirements 
were  defined.  Further,  it  faced  the  risk  that  insufficient  funding  would 
impact  the  program  office’s  ability  to  acquire  NMCI  assets.  Nevertheless, 
the  program  was  approved  to  proceed. 


33 At  the  time  of  the  Gate  4  review,  DON  was  using  the  Probability  of  Program  Success 
version  1  framework  that  consisted  of  17  metrics.  Technology  Protection  was  added  in 
version  2  of  the  framework. 
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Figure  4:  NGEN  Gate  4  Probability  of  Program  Success  Assessment  (November 
2009) 
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Source:  GAO  analysis  of  DON  data. 


Note:  Green  means  that  the  program  is  on  track  to  provide  capability  within  approved  costand 
schedule  constraints;  yellow  (pattern)  means  that  the  program  has  identified  some  significant  issues 
with  providing  capability  within  approved  costand  schedule  constraints,  but  mitigation  strategies  are 
being  executed;  red  means  thatthe  program  has  identified  significant  issues  thatwill  inhibit  delivery 
of  capability  within  approved  costand  schedule  constraints  and  mitigation  strategies  have  not  been 
identified;  and  "critical'’ means  that  a  potential  nonexecutable  situation  exists  for  the  program  that 
must  be  remedied. 


“These  metrics  are  considered  critical  when  they  are  scored  red. 

Gate  5  review.  For  this  review,  which  was  conducted  in  October  2010, 
DON  again  used  its  Probability  of  Program  Success  methodology  and 
assessed  program  performance  and  risk  against  all  18  metrics,  including  9 
that  DON  designated  as  potentially  critical.  Three  metrics  were  rated  as 
red;  1,  test  and  evaluation,  was  deemed  critical.  According  to  the 
assessment,  the  test  and  evaluation  metric  was  rated  as  red  because  the 
Test  and  Evaluation  Master  Plan  was  not  complete;  the  budget  and 
planning  metric  was  rated  as  red  because  of  significant  NGEN  funding 
reductions;  and  the  manning  metric  was  rated  as  red  because  of 
inadequate  program  office  contracting,  engineering  and  logistics 
personnel.  Further,  according  to  the  assessment,  the  Test  and  Evaluation 
Master  Plan  was  not  complete  because  the  requirements  were  not  defined. 
As  discussed  above,  the  program  recognized,  at  Gate  4,  the  risk  that  a 
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delay  in  defining  NGEN  requirements  would  impact  the  completion  of  this 
plan.  (See  fig.  5  for  the  assessment  results  for  all  18  metrics.)  According  to 
the  gate  briefing  document,  these  red  ratings  introduced  a  number  of  risks, 
such  as  the  risk  that  the  program  would  not  be  able  to  execute  its  current 
acquisition  approach  and  meet  program  milestones. 

In  addition,  even  though  the  assessment  rated  the  acquisition  management 
metric  as  green,  this  rating  is  not  consistent  with  our  findings  in  this  report 
about  the  NGEN  integrated  master  schedule.  Specifically,  the  rationale  for 
the  green  rating  was  that  the  August  2010  rebaselined  schedule  was 
viewed  as  realistic  and  achievable  by  key  stakeholders.  However,  as  stated 
earlier,  program  officials  conceded  that  the  schedule  does  not  assign 
resources,  and  the  scheduling  tool  is  unable  to  generate  a  valid  critical 
path,  which  are  key  scheduling  practices;  thus  the  August  2010  schedule 
was  not  reliable.  The  approval  of  the  Assistant  Secretary  of  the  Navy 
(Research,  Development  and  Acquisition)  for  NGEN  to  proceed  beyond 
Gate  5  was  made  conditional  on  the  program  satisfactorily  completing 
action  items  focused  on  releasing  the  request  for  proposals  for  the 
Transport  Services  contract  (scheduled  for  January  2011)  and  resolving  its 
funding  shortfall. 
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Figure  5:  NGEN  Gate  5  Probability  of  Program  Success  Assessment  (October  2010) 
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Source:  GAO  analysis  of  DON  data. 


“These  metrics  are  considered  critical  when  certain  underlying  criteria  are  rated  red.  This  approach  is 
different  from  the  Gate  4  assessment  because  DON  used  a  revised  version  of  the  P  robability  of 
Program  Success  framework  when  it  conducted  the  Gate  5  assessment. 


As  shown  above,  DON  has  demonstrated  a  pattern  of  approving  NGEN  at 
key  acquisition  review  gates  in  the  face  of  both  limited  disclosure  of  the 
program’s  health  and  risks  and  known  program  risks  and  shortfalls  in 
performance.  According  to  DON  officials,  the  decisions  to  pass  the  gates 
and  proceed  were  based  on  their  view  that  they  had  sufficiently  mitigated 
known  risks  and  issues.  By  not  fully  ensuring  that  NGEN  gate  decisions 
sufficiently  reflected  program  challenges,  DON  has  increased  the 
likelihood  that  the  NGEN  acquisition  alternative  that  it  is  pursuing  is  not 
the  most  cost-effective  course  of  action,  and  that  the  program  will  cost 
more  and  take  longer  to  complete  than  planned. 


Conclusions 


Given  the  enormous  size,  complexity,  and  mission  importance  of  NGEN,  it 
is  vital  that  DON  and  DOD  assure  decision  makers,  including  the 
congressional  defense  committees,  that  the  approach  to  acquiring  needed 
capabilities  is  the  most  cost-effective  and  that  its  execution  is  guided  by  a 
well-defined  schedule  and  informed  milestone  decision  making.  To  date, 
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this  has  not  occurred  to  the  degree  that  it  should.  Most  notably,  while 
DON  produced  substantially  well-documented  cost  estimates,  the  NGEN 
acquisition  approach  currently  being  followed  is  not  grounded  in  a  reliable 
analysis  of  alternative  approaches,  and  the  selected  approach  was  not 
even  assessed  and  is  about  $4.7  billion  costlier  and  introduces  more  risk 
than  the  alternatives  that  were  assessed.  Further,  the  program’s  execution 
to  date  has  not  been  based  on  the  kind  of  reliably  derived  integrated 
master  schedule  that  is  essential  to  program  success.  While  the  program 
office  is  aware  of  some  of  the  schedule  weaknesses  and  intends  to  address 
them,  additional  work  is  needed  to  ensure  that  the  schedule  can  produce 
credible  dates  for  planned  NGEN  milestones  and  events.  Exacerbating  this 
is  an  equally  troubling  pattern  of  missed  milestones  and  delays  in  key 
program  documentation,  as  well  as  gate  review  decisions  that  have 
allowed  the  program  to  proceed  in  the  face  of  significant  performance 
shortfalls  and  risks. 

While  NGEN  is  scheduled  for  an  OSD-level  milestone  review  in  August 
2011,  the  above  schedule  limitations  make  it  likely  that  this  review  date 
will  slip.  It  is  thus  imperative,  given  the  scope  and  nature  of  the  program’s 
problems,  that  swift  and  immediate  action  be  taken  to  ensure  that  the 
most  cost-effective  acquisition  approach  is  pursued  and  that  a  reliable 
schedule  and  performance-  and  risk-based  decision  making  are  employed. 
To  do  less  increases  the  chances  that  needed  NGEN  capabilities  will  be 
delivered  late  and  be  more  costly  than  necessary. 


Recommendations  for 
Executive  Action 


To  ensure  that  NGEN  capabilities  are  acquired  in  the  most  cost-effective 
manner,  we  recommend  that  the  Secretary  of  Defense  take  the  following 
two  actions: 


•  direct  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  to  conduct  an  interim  NGEN  milestone  review,  and 

•  direct  the  Secretary  of  the  Navy  to  immediately  limit  further  investment  in 
NGEN  until  this  review  has  been  conducted  and  a  decision  on  how  best  to 
proceed  has  been  reported  to  the  Secretary  of  Defense  and  congressional 
defense  committees. 

At  a  minimum,  this  review  should  ensure  that  DON  pursues  the  most 
advantageous  acquisition  approach,  as  evidenced  by  a  meaningful  analysis 
of  all  viable  alternative  acquisition  approaches,  to  include  for  each 
alternative  reliably  derived  cost  estimates  and  metrics-based  operational 
effectiveness  analyses.  In  addition,  the  review  should  consider  existing 
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performance  shortfalls  and  known  risks,  including  those  discussed  in  this 
report. 

To  facilitate  implementation  of  the  acquisition  approach  resulting  from  the 
above  review,  we  further  recommend  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy  to  take  the  following  two  actions: 

•  ensure  that  the  NGEN  integrated  master  schedule  substantially  reflects  the 
key  schedule  estimating  practices  discussed  in  this  report,  and 

•  ensure  that  future  NGEN  gate  reviews  and  decisions  fully  reflect  the  state 
of  the  program’s  performance  and  its  exposure  to  risks. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  signed  by  the  Deputy 
Assistant  Secretary  of  Defense  (C3,  Space  and  Spectrum),  and  reprinted  in 
appendix  II,  DOD  stated  that  it  concurred  with  one  of  our  four 
recommendations,  did  not  concur  with  one  recommendation,  and  partially 
concurred  with  two.  The  department’s  comments  are  discussed  below. 

The  department  partially  concurred  with  our  recommendation  to  conduct 
an  interim  milestone  review  that  provides  assurance  that  DON  is  pursuing 
the  most  advantageous  acquisition  approach.  Specifically,  the  department 
stated  that  it  intended  to  leverage  the  next  OSD-chaired  NGEN 
Overarching  Integrated  Product  Team  meeting  in  February  2011  for  the 
review  and  that  following  this  meeting,  the  USD(AT&L)  will  conduct  a 
Milestone  Decision  Authority  review  of  the  current  NGEN  approach,  along 
with  risks.  According  to  the  department,  this  approach  balances  the 
review  processes  already  in  place,  resource  constraints,  and  the  need  for 
an  additional  milestone  review.  Further,  the  department  said  it  had 
concluded  that  DON’s  AOA  was  sufficient  and  that  the  analysis  had  been 
approved  by  CAPE.  DOD  added  that  it  will  complete  an  economic 
analysis — a  post  AOA-activity — for  the  August  2011  Milestone  C  review, 
which  will  include  a  follow-on  independent  cost  estimate  and  an  updated 
determination  of  the  most  cost-effective  solution. 

While  these  are  important  steps,  DOD’s  planned  actions  do  not  appear  to 
fully  address  our  recommendation.  Specifically,  the  department  did  not 
indicate  any  intent  to  reevaluate  whether  the  current  solution  is  indeed  the 
most  advantageous  approach,  despite  the  weaknesses  contained  in  the 
AOA  identified  in  this  report  and  the  fact  that  the  current  approach  was 
not  included  in  its  analysis.  According  to  the  September  2010  draft  NGEN 
economic  analysis  development  plan,  only  the  status  quo  and  the  current 
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approach  are  to  be  analyzed,  not  the  other  three  alternatives  that  were 
included  in  the  AO  A.  Without  a  meaningful  analysis  of  alternatives,  DOD 
will  be  unable  to  determine  the  most  cost-effective  solution  in  its  two 
upcoming  key  reviews. 

•  The  department  did  not  concur  with  our  recommendation  that  it  limit 
further  investment  in  NGEN  until  a  decision  has  been  made  on  how  best  to 
proceed  based  on  an  interim  review  that  considers  all  viable  alternative 
acquisition  approaches  and  this  decision  has  been  reported  to  the 
Secretary  of  Defense  and  to  congressional  defense  committees.  The 
department  stated  that  DON’s  NGEN  acquisition  strategy  and  program 
management  have  been  approved  by  the  milestone  decision  authority,  and 
that  adequate  oversight  is  in  place  to  ensure  regulatory  and  statutory 
compliance.  Further,  the  department  said  that  the  limitation  on  NGEN 
investments  will  impact  future  DON  business  operations  and,  ultimately, 
Naval  warfighting  capabilities.  The  department  added  that  it  will  make 
adjustments  to  NGEN  investments  if  it  determines  they  are  required; 
however,  it  also  said  it  must  continue  to  execute  the  investments  within 
the  time  frame  of  the  continuity  of  services  contract. 

While  oversight  is  in  place  for  the  NGEN  program,  it  is  not  effective. 
Specifically,  as  discussed  in  this  report,  DON’s  past  reviews  have  resulted 
in  decisions  that  were  not  always  performance-  and  risk-based.  Given  that 
DON  is  continuing  to  proceed  in  the  face  of  the  problems  we  are  reporting, 
it  is  even  more  important  that  adequate  oversight  be  provided  by  the 
Secretary  and  congressional  defense  committees. 

Moreover,  we  maintain  that  limiting  further  investment  in  NGEN,  thereby 
delaying  the  Milestone  C  event  and  its  associated  activities,  is  the  most 
prudent  action  at  this  time.  By  not  evaluating  all  viable  acquisition 
approaches  before  proceeding  with  further  investment  in  NGEN,  the 
department  cannot  be  assured  that  it  is  pursuing  the  most  cost-effective 
approach.  Further,  by  selecting  an  approach  that,  as  discussed  in  this 
report,  carries  greater  relative  schedule  and  performance  risks  than  other 
alternatives  and  is  being  executed  against  an  unreliable  program  schedule, 
the  department  increases  the  risk  that  its  approach  will  lead  to  future  cost 
overruns,  requiring  it  to  expend  additional  resources  that  could  otherwise 
be  used  to  provide  other  warfighting  capabilities.  Furthermore,  even  if  the 
department  proceeds  along  its  current  course,  the  issues  we  have 
identified  with  the  program’s  schedule,  along  with  the  delays  already 
experienced,  raise  concerns  that  it  will  be  unable  to  complete  the 
transition  as  planned  within  the  time  frames  of  the  current  continuity  of 
services  contract. 
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•  The  department  partially  concurred  with  our  recommendation  that  the 
Secretary  of  Defense  direct  the  Secretary  of  the  Navy  to  ensure  that  the 
NGEN  integrated  master  schedule  substantially  reflects  the  key  schedule 
estimating  practices  discussed  in  this  report.  DOD  stated  that  the 
integrated  master  schedule  was  developed  in  accordance  with  industry 
best  practices.  However,  as  discussed  in  this  report,  none  of  the 
subschedules  that  we  analyzed  reflected  all  the  practices  that  our  work 
has  identified  as  necessary  to  develop  and  maintain  a  reliable  schedule.  To 
its  credit,  DOD  also  said  it  would  seek  ways  to  improve  schedule 
performance  and  that  DON  will  review  the  scheduling  practices  discussed 
in  this  report  and  incorporate  those  found  to  be  beneficial.  We  continue  to 
believe  that  the  Secretary  of  the  Navy  should  ensure  that  the  NGEN 
integrated  master  schedule  incorporates  all  of  the  best  practices  for 
schedule  estimating  discussed  in  this  report  to  help  manage  and  measure 
its  progress  in  executing  the  work  needed  to  proceed  through  Milestone  C 
and  ultimately  transition  from  NMCI  to  NGEN. 

•  The  department  concurred  with  our  recommendation  to  ensure  that  future 
NGEN  gate  reviews  and  decisions  fully  reflect  the  state  of  the  program’s 
performance  and  its  exposure  to  risks.  In  this  regard,  the  department 
stated  that  it  plans  to  continue  to  conduct  monthly  risk  management  board 
meetings  and  program  health  reviews,  and  report  the  results  to  program 
leadership.  It  will  be  critical  that  decisions  on  NGEN  fully  reflect  the  state 
of  the  program’s  performance  and  exposure  to  risks. 
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We  are  sending  copies  of  this  report  to  the  appropriate  congressional 
committees;  the  Director,  Office  of  Management  and  Budget;  the 
Congressional  Budget  Office;  the  Secretary  of  Defense;  and  the  Secretary 
of  the  Navy.  The  report  also  is  available  at  no  charge  on  the  GAO  Web  site 
at  http://www.gao.gov. 

If  you  or  your  staff  members  have  any  questions  on  matters  discussed  in 
this  report,  please  contact  me  at  (202)  512-6304  or  melvinv@gao.gov. 
Contact  points  for  our  Offices  of  Congressional  Relations  and  Public 
Affairs  may  be  found  on  the  last  page  of  this  report.  GAO  staff  who  made 
major  contributions  to  this  report  are  listed  in  appendix  III. 

Valerie  C.  Melvin 

Director,  Information  Management 
and  Human  Capital  Issues 
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Our  objectives  were  to  determine  whether  (1)  the  Department  of  the  Navy 
(DON)  sufficiently  analyzed  alternative  approaches  for  acquiring  its  Next 
Generation  Enterprise  Network  (NGEN),  (2)  DON  has  a  reliable  program 
schedule  for  executing  NGEN,  and  (3)  acquisition  decisions  have  been 
performance-  and  risk-based. 

To  address  the  first  objective,  we  evaluated  the  analysis  of  alternatives 
(AO A)  report  and  its  supporting  documentation  against  relevant 
Department  of  Defense  (DOD)  guidance1  and  GAO’s  Cost  Estimating  and 
Assessment  Guide2  and  compared  the  alternatives  in  the  AOA  final  report 
with  the  NGEN  Acquisition  Strategy.  More  specifically, 

•  For  the  cost  analysis,  we  compared  the  AOA  cost  estimating 
documentation,  such  as  the  cost  model  spreadsheet,  supporting 
documentation  for  the  cost  model,  and  the  final  NGEN  AOA  report, 
against  the  four  characteristics  of  a  reliable  estimate3  in  GAO’s  Cost 
Estimating  and  Assessment  Guide  to  determine  the  extent  to  which  the 
cost  estimates  reflected  each  of  the  four  characteristics. 

•  For  the  operational  effectiveness  analysis,  we  compared  an  NGEN 
alternatives  performance  assessment  report  and  the  AOA  final  report 
against  the  relevant  DOD  guidance  to  determine  the  extent  to  which  the 
analysis  was  sufficient.  In  addition,  we  reviewed  NGEN  AOA  Advisory 
Group  meeting  minutes  and  documentation  containing  the  results  of  a 
Space  and  Naval  Warfare  Systems  Command  review  of  the  cost  analysis. 
We  also  interviewed  cognizant  DON  and  Office  of  the  Secretary  of  Defense 
officials  about  the  AOA’s  development  and  results. 

To  address  the  second  objective,  we  first  reviewed  the  integrated  master 
schedule  and  4  of  the  29  subschedules  that  existed  when  we  began  our 
review  and  that  comprised  the  early  transition  activities  intended  to 
address  key  program  risks,  as  well  as  high-level  plans  for 
postdeployment.4  Accordingly,  we  focused  on  assessing  the  May  2010 


'Defense  Acquisition  University,  Defense  Acquisition  Guidebook,  Section  3.3  “Analysis  of 
Alternatives”  (accessed  Mar.  19,  2010);  DON  Acquisition  and  Capabilities  Guidebook 
(December  2008);  and  Air  Force  Analysis  of  Alternatives  Handbook  (July  2008). 

2GAO-09-3SP. 

3These  are  (1)  comprehensive,  (2)  well-documented,  (3)  accurate,  and  (4)  credible. 

4These  schedules  were  the  Transition  Integrated  Product  Team,  IT  Service  Management 
Process  Development,  Comprehensive  Facilities  Inventory  Phase  2  Plan  of  Action  and 
Milestones,  and  Contract  Technical  Representative  Workforce  Reconstitution. 
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subschedules  against  the  nine  key  schedule  estimating  practices6  in  GAO’s 
Cost  Estimating  and  Assessment  Guide  using  commercially  available 
software  tools  to  determine  the  extent  to  which  each  subschedule 
reflected  each  of  the  nine  practices  (e.g.,  a  logical  sequence  of  activities 
and  reasonable  activity  durations).  Further,  we  characterized  the  extent  to 
which  each  subschedule  satisfied  each  of  the  practices  as  either  met, 
substantially  met,  partially  met,  minimally  met,  or  not  met.6  In  addition,  we 
compared  the  baseline  schedule,  established  in  December  2009,  to  the 
rebaselined  schedule,  established  in  August  2010,  to  identify  whether  key 
event  and  milestone  dates  had  slipped.  We  also  interviewed  cognizant 
officials  about  development  and  management  of  the  integrated  master 
schedule  and  underlying  subschedules.  We  also  reviewed  program 
documentation,  such  as  the  NGEN  schedule  management  plan,  program 
performance  reports,  program  management  reviews,  and  the  acquisition 
strategy. 

To  address  the  third  objective  we  compared  program  review 
documentation,  such  as  briefings,  program  performance  assessments,  and 
meeting  minutes,  to  DON  acquisition  review  policies  and  procedures,  as 
well  as  to  other  programmatic  documents,  such  as  risk  registers  and  risk 
management  board  briefings  and  meeting  minutes.  We  also  interviewed 
cognizant  program  officials  regarding  NGEN  performance  and  program 
risks. 

To  assess  the  reliability  of  the  data  that  we  used  to  support  the  findings  in 
this  report,  we  reviewed  relevant  program  documentation  to  substantiate 
evidence  obtained  through  interviews  with  agency  officials.  We 
determined  that  the  data  used  in  this  report  are  sufficiently  reliable.  We 
have  also  made  appropriate  attribution  indicating  the  sources  of  the  data. 


5These  are  (1)  capturing  all  activities,  (2)  sequencing  all  activities,  (3)  assigning  resources 
to  all  activities,  (4)  establishing  the  duration  of  ah  activities,  (5)  integrating  activities 
horizontally  and  vertically,  (6)  establishing  the  critical  path  for  ah  activities,  (7)  identifying 
reasonable  “float”  between  activities,  (8)  conducting  a  schedule  risk  analysis,  and  (9) 
updating  the  schedule  using  logic  and  durations. 

6“Met”  means  DON  provided  complete  evidence  that  satisfies  the  entire  criterion. 
“Substantially  Met”  means  DON  provided  evidence  that  satisfies  a  large  portion  of  the 
criterion.  “Partially  Met”  means  DON  provided  evidence  that  satisfies  about  half  of  the 
criterion.  “Minimally  Met”  means  DON  provided  evidence  that  satisfies  a  small  portion  of 
the  criterion.  “Not  Met”  means  DON  provided  no  evidence  that  satisfies  any  of  the 
criterion. 
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We  conducted  this  performance  audit  at  DOD  offices  in  the  Washington, 
D.C.,  metropolitan  area  and  at  the  Space  and  Naval  Warfare  Systems 
Command  in  San  Diego,  California,  from  October  2009  to  February  2011  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 
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OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 

6000  DEFENSE  PENTAGON 
WASHINGTON,  DC.  20301-6000 


fEB  1  5  Mil 


Ms.  Valerie  C.  Melvin 

Director,  Information  Management  and  Human  Capital  Issues 
U.S.  Government  Accountability  Office 
441  G  Street,  NW 
Washington,  DC  20548 

Dear  Ms.  Melvin: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report,  GAO-1 1- 
150,  ‘INFORMATION  TECHNOLOGY:  Better  Informed  Decision  Making  Needed  on  Navy’s 
Next  Generation  Enterprise  Network  Acquisition,’  dated  December  15,  2010  (GAO  Code 
3 1 0679).  We  acknowledge  receipt  of  the  draft  report  and  note  that  the  Department’s  written 
comments  are  included. 


Our  point  of  contact  is  Mr.  Mark  Godino  at  (703)  602-2720,  ext  142.  Please  contact  him 
with  any  questions  or  if  you  need  clarification. 

Sincerely, 


Dr.  Ronald  C.  Jost 

Deputy  Assistant  Secretary  of  Defense 
(C3,  Space  and  Spectrum) 


Enclosure: 
As  Stated 
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GAO  DRAFT  REPORT  DATED  DECEMBER  15,  2010 
GAO-11-150  (GAO  CODE  310679) 

“INFORMATION  TECHNOLOGY:  BETTER  INFORMED  DECISION 
MAKING  NEEDED  ON  NAVY’S  NEXT  GENERATION  ENTERPRISE 
NETWORK  ACQUISITION” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense  direct  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (USD(AT&L))  to 
conduct  an  interim  Next  Generation  Enterprise  Network  (NGEN)  milestone  review. 

DOD  RESPONSE:  Partially  Concur.  The  Department  concurs  with  the  finding  that  an 
interim  milestone  review  may  provide  additional  benefit,  even  though  a  thorough  review  was 
conducted  by  the  Office  of  the  Secretary  of  Defense  (OSD)  chaired,  NGEN  Overarching 
Integrated  Product  Team  (OIPT)  in  November  201 0.  The  Department  intends  to  further  leverage 
the  next  upcoming  OSD  chaired  NGEN  OIPT  scheduled  for  17  February  201 1,  for  this  purpose. 
Following  this  OIPT  meeting,  the  USD(AT&L)  will  conduct  a  Milestone  Decision  Authority 
(MDA)  review  of  the  current  NGEN  acquisition  approach,  along  with  the  risks.  This  approach 
reconciles  the  balance  between  recognition  of  vetted  consensus  review  processes  already  in 
place,  resource  constraints,  and  the  need  for  an  additional  milestone  review.  The  Department  has 
concluded  the  Department  of  the  Navy  (DON)  Analysis  of  Alternatives  (AoA)  was  sufficient. 
The  Office  of  the  Director  of  Cost  Assessment  and  Program  Evaluation  (CAPE)  approved  the 
NGEN  Increment  1  AoA  in  December  2009.  Per  Clinger-Cohen  Certification  and  DoD 
requirements,  a  follow-on  independent  cost  estimate  and  an  updated  determination  of  the  most 
cost  effective  solution  are  completed  as  part  of  the  Economic  Analysis.  The  Economic  Analysis 
is  a  post-AoA  activity  and  will  be  completed  at  Milestone  C,  when  the  NGEN  program  is 
baselined  (MDA  approves  Acquisition  Program  Baseline  (APB))  in  August  2011. 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense  direct  the 
Secretary  of  the  Navy  to  immediately  limit  further  investment  in  NGEN  until  this  review  has 
been  conducted  and  a  decision  on  how  best  to  proceed  has  been  reported  to  the  Secretary  and 
Congressional  defense  committees. 

DOD  RESPONSE:  Nonconcur.  The  DON'S  Next  Generation  Enterprise  Network  (NGEN) 
program  management  and  investment  strategy  is  well  defined  in  its  Acquisition  Strategy,  which 
was  approved  by  the  MDA  and  further  clarified  in  subsequent  Acquisition  Decision 
Memorandums  (ADMs).  Additionally,  the  NGEN  program  has  definitive,  dedicated,  and 
qualified  acquisition  oversight  in  place  to  ensure  its  regulatory  and  statutory  compliance.  The 
DON  is  presently  in  the  midst  of  a  limited-period  Continuity  of  Service  Contract  (CoSC)  that  is 
providing  bridge  services  during  the  transition  from  the  present  Navy  Marine  Corps  Intranet 
(NMCI)  network  to  the  future  NGEN  state.  If,  during  the  upcoming  interim  review  by  the 
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USD(AT&L),  the  Department  feels  adjustments  are  required  to  the  current  or  planned  NGEN 
investments,  actions  will  be  taken  at  that  time.  Limitation  of  NGEN  investments,  at  this  time, 
will  significantly  impact  future  DON  business  operations  and  ultimately.  Naval  warfighting 
capabilities.  In  the  near  term,  such  limitations  would  impair  Program  Management  Office 
activities  critical  to  support  planned  Milestone  C  events  in  the  Fourth  Quarter  of  FY 1 1 .  Timely 
execution  of  NGEN  investments  must  continue  in  order  to  successfully  transition  from  CoSC 
within  the  contractually  obligated  timeframe  allotted. 

RECOMMENDATION  3:  To  facilitate  implementation  of  the  acquisition  approach  resulting 
from  the  above  review,  the  GAO  recommends  that  the  Secretary  of  Defense  direct  the  Secretary 
of  the  Navy  to  ensure  that  the  (NGEN)  integrated  master  schedule  substantially  reflects  the  key 
schedule  estimating  practices  discussed  in  this  report. 

DOD  RESPONSE:  Partially  Concur.  The  NGEN  Integrated  Master  Schedule  (IMS)  was 
developed  in  accordance  with  industry  best  practices  for  scheduling  and  was  recently  finalized 
and  approved  by  the  Program  Manager  (PM)  on  26  August  2010.  The  NGEN  IMS  will  continue 
to  evolve  as  we  move  forward  to  Milestone  C  and  an  approved  Acquisition  Program  Baseline 
(APB).  As  the  NGEN  IMS  evolves,  the  Department  will  continue  to  seek  out  ways  to  improve 
our  schedule  performance.  The  Department  will  review  the  key  schedule  estimating  practices 
discussed  in  this  report  and  will  consider  incorporating  those  found  to  be  beneficial. 

RECOMMENDATION  4:  To  facilitate  implementation  of  the  acquisition  approach  resulting 
from  the  above  review,  the  GAO  recommends  that  the  Secretary  of  Defense  direct  the  Secretary 
of  the  Navy  to  ensure  that  future  NGEN  gate  reviews  and  decisions  fully  reflect  the  state  of  the 
program’s  performance  and  its  exposure  to  risks. 

DOD  RESPONSE:  Concur.  The  NGEN  PM  will  continue  to  conduct  monthly  formal  Risk 
Management  Board  (RMB)  and  Probability  of  Program  Success  (PoPS)  (V2.0)  reviews.  Results 
of  both  of  these  formal  meetings  are  regularly  reported  in  the  NGEN  Program  Dashboard,  which 
is  distributed  weekly  to  program  leadership  and  monthly  to  the  Navy  Senior  Integration  Board 
(NSIB)  chaired  by  Principal  Civilian  Deputy  Assistant  Secretary  of  the  Navy  (Research, 
Development,  and  Acquisition)  (PCDASN(RDA)).  Additionally,  program  risk  status  is  briefed 
at  the  OSD  chaired  OIPT  meetings.  Finally,  both  past  and  future  NGEN  gate  reviews  have  and 
will  continue  to  fully  reflect  the  state  of  the  program’s  performance  and  its  exposure  to  risk. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 

MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 


Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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